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Introduction 


About this evidence guide 


This evidence guide provides detailed research summaries on 
the effectiveness of specific afterschool programs for improving 
outcomes for students in grades K-12. The purpose of the guide 
is to present information that education agencies, afterschool 
providers, and others can use as they make decisions about 
afterschool programming. 


The guide is a companion to Afterschool Programs: A Review 

of Evidence Under the Every Student Succeeds Act, which is 
available for free download on the Research for Action website 
https://www.researchforaction.org/projects/afterschoolessa/. The 
review is based on a comprehensive, systematic literature search 
for studies published between 2000-17 of the effectiveness of 
afterschool programs. 


Why this guide? 

The afterschool field and its evaluators have invested 
considerable effort in evaluating whether afterschool programs 
make a difference for students—that is, whether they “work.” 
Prior to this review, the full extent of evidence-building about 
afterschool programs was not known. This guide demonstrates 
that, since 2000, there have been many rigorous studies of the 
effectiveness of afterschool programs. Further, there are numer- 
ous afterschool programs with evidence of improving students’ 
academic skills, school attendance, physical health, and other 
outcomes. Not every program improves outcomes, but when 
programs are studied using rigorous research designs, even 
those with no effect, mixed effects, or negative effects contribute 
important learning to the field about what works, what doesn’t, 
where, and for whom. 


The value of investments in building evidence about afterschool 
programs became especially clear when the Every Student 
Succeeds Act, or ESSA, became federal law in 2015. ESSA 
articulates an evidence framework and encourages (or in some 
cases, requires) its use in programs authorized under the law. 
Several funding sources for afterschool programs—most notably, 
the 21st Century Community Learning Centers program, funded 
at over $1 billion annually—are authorized under ESSA (see Box 
1 for a summary of ESSA titles and evidence requirements). This 
guide uses the ESSA evidence framework, described below, to 
assess the evidence for a wide array of afterschool programs. 


This guide provides information to use in decision-making about 
afterschool programming. To use evidence wisely and well, decision- 
makers should consider whether a program has evidence of 
effectiveness—as well as other characteristics that make it a good 
fit. Guidance from the U.S. Department of Education on applying 
ESSA’s evidence framework encourages education agencies 

to select approaches based on their evidence of effectiveness 
and a thorough assessment of community needs, resources, 

and priorities. Programs must also be feasible, affordable, and 
appropriate for the context and the students they will serve. 


This guide also can be used to identify where evidence of 
effectiveness is still needed. If you know of an afterschool 
program that is not included in this guide, it means that we 
were not able to identify any studies of the program’s impact 
on common school-related or physical activity/health outcomes 
that met our ESSA-informed definition of strong effectiveness 
research, despite a comprehensive literature search. ESSA 
encourages education agencies and program providers to fill 
the evidence gaps by carrying out well-designed studies of 
afterschool programs. Likewise, the field can continue to advance 
by conducting more rigorous studies of programs that already 
have less-rigorous studies of their effectiveness. 
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What this evidence guide does not do 


This evidence guide is intended to help decision-makers identify effective afterschool 


programs. For this reason, the guide and the evidence review on which it is based do not 


specifically address the larger question of whether afterschool programs, as a class, are 


effective for improving student outcomes. Further, the review does not rigorously examine 


which parts of individual programs were effective or seek to generalize across programs 


about the effectiveness of specific components. 


ESSA’s framework for evidence of program effectiveness 


ESSA’s evidence framework has four levels, or tiers, ranging from the most rigorous evidence 


of effectiveness (Tier |, or Strong) to the least rigorous (Tier IV, or Demonstrates a Rationale). 


In this guide, we summarize evidence that meets Tier I-Ill research design requirements for 


establishing a cause-and-effect relationship between the program and student outcomes. It 


is important to note that studies with a rigorous research design do not necessarily show that 


a program improves outcomes. This guide includes studies regardless of whether they show 


improved outcomes because it is also important to know when a program does not have 


hoped-for effects. Studies that could meet Tier IV requirements are summarized in Appendix 


EG-2 of this evidence guide. 


ESSA evidence tiers and their requirements for establishing a cause-and-effect relationship 


are briefly summarized in Box 1. More information about the definition of these tiers is 


provided in Chapter 2 of the evidence review. 


What this guide includes 


This guide includes several types of evidence summaries. 


« Summaries of evidence for afterschool programs. 


For each program with at least one study that meets Tier 


I-Ill research quality standards under ESSA (described 


below), we provide a multi-page program summary. This 


summary includes information about program effects, 


where and with whom the research was conducted, and 


basic facts about the program’s implementation. 


- Brief summaries of the effectiveness of school-sponsored 


extracurricular programs (Appendix EG-1). Studies 


of the effects of participation in extracurricular programs 


are within the scope of this review, but because these 
studies typically provide little information about the 
content of the programming, we provide paragraph- 
length summaries only. 


Brief summaries of studies that meet a lower research 
quality standard (Appendix EG-2). For studies that 
compare outcomes for program participants and non- 
participants but do not meet all Tier I-Ill research quality 
requirements, we provide a paragraph about the study and 
its findings. These studies contribute to a full accounting 
of the effectiveness studies identified in this review. 


Brief summaries of studies with insufficient information 
to assign an ESSA tier (Appendix EG-3). These studies, as 
reported, were missing information needed to determine 
an ESSA rating. These studies contribute to a full accounting 
of the effectiveness studies identified in this review. 


Brief summaries of studies of year-round out-of- 

school time programs (Appendix EG-4). Studies of 
programs that combine afterschool and summer learning 
programming are not within the scope of this review but 
may be of interest to decision-makers. 
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Using ESSA evidence tiers in practice: Additional recommendations 


To help states apply ESSA evidence tiers to activities funded under the law, in 2016 the U.S. 


Department of Education issued non-binding, non-regulatory guidance for using the evidence 


tiers in practice. This guidance, paraphrased below, recommends that decision-makers 


consider several features of the evidence beyond a cause-and-effect research design and 


at least one statistically significant improved outcome. Decision-makers should consider 


whether the evidence demonstrates: 


Broad application. That is, the program has demonstrated its effectiveness in research 


with a sufficiently large group of students (at least 350 students) and in multiple places 


(at least two school districts). 


Similarity. The program has improved outcomes for similar students and/or in a context 


similar to where the program would be implemented. 


Overall effectiveness. The whole body of evidence about the program indicates that the 


program improves relevant outcomes. 


No harm. There are no negative findings that would cast doubt on the overall benefit of 


the program for students. 


These recommendations for making sense of research evidence, along with the law’s 


requirements, are described in more detail in Chapter 2 of the evidence review. 


The program summaries in this evidence guide provide the information needed to examine 


whether the evidence for each program meets these additional recommendations. 


BOX 1 


Evidence Tiers in ESSA 


Programs with Tier | evidence must be supported by 

at least one experimental study, the “gold standard” for 
establishing cause-and-effect relationships. In these 
studies, students are randomly assigned to experience 

a program or to the control group. The study must show 
that the program improved at least one outcome, and the 
improvement must be statistically significant, or unlikely to 
be the result of chance variation. 


Programs with Tier Il evidence must be supported by at 
least one quasi-experimental study that compares out- 
comes for afterschool program participants to outcomes for 
a comparison group that is closely matched on important 
characteristics. As with Tier | evidence, the study must 
show that the program improved at least one outcome, and 
the improvement must be statistically significant. 


Programs with Tier Ill evidence must be supported by at 
least one study that the law describes as “correlational... 
with statistical controls for selection bias.” Although not 
specified in the law, the implication is that Tier Il and Tier Ill 
studies have many similarities but program and comparison 
groups in Tier Ill studies are not as closely matched. For 
example, compared to Tier II studies, Tier Ill studies may 
have larger differences between the program and compar- 
ison groups on previous achievement, which raises more 
doubt about whether the study represents 

an “apples-to-apples” comparison. 


Programs that meet Tier IV requirements provide a 
rationale for why outcomes are likely to improve based on 
existing research described only as “high-quality” in the 
law and are undergoing evaluation of their effectiveness. 


How we report overall effectiveness of programs 
To help readers understand the overall program effectiveness, 
as recommended by the Department guidance, we provide 
summaries for each outcome domain. An outcome domain is 
a broad category of similar outcome measures. Examples of 
outcome domains are (1) Reading/ELA Achievement and (2) 
School Attendance and Enrollment. 


For each outcome domain in a study, we provide a statement of 
the overall consistency and direction of effects, using one of four 
descriptors: 


« Positive Effect. The study found at least one improved 
outcome and no overriding contrary evidence. 


« Mixed Effects. The study found a mix of improved and null 
or negative outcomes. 


- No Effect. The study found neither improved nor negative 
outcomes. 


« Negative Effect. The study found at least one negative 
outcome with no overriding contrary evidence. 


BOX 2 


Definitions of overall effectiveness 


To summarize the overall effectiveness of programs for improving 
specific outcome domains, this review uses the following 
definitions. These definitions are adapted from the criteria used 
by the What Works Clearinghouse to determine effectiveness 
ratings (see What Works Clearinghouse Procedures Handbook v. 
4.0, Table IV.3). 


Positive Effect. In this review, an outcome domain is described as 
demonstrating a positive effect when: 


« There is only one outcome in the domain, and that outcome 
shows a Statistically significant improvement, OR 


« There are at least two statistically significant improved out- 
comes and no statistically significant negative outcomes. 


Mixed Effects. In this review, an outcome domain is described as 
demonstrating mixed effects when: 


- There is at least one statistically significant improved 
outcome and at least one statistically significant negative 
outcome, but there are more improved than negative 
outcomes OR 


To contribute to an outcome domain descriptor, the outcome 
must meet Tier I-IIl requirements for establishing a cause-and- 
effect relationship. When there is more than one study of a 
program, we provide these descriptors for all studies of the 
program combined. More detail about how these descriptors are 
defined appears in Box 2. 


How required evidence varies by ESSA program 


Under some sections of ESSA, one or more activities must be 
supported by evidence of effectiveness at Tiers I-III. For other 
ESSA programs, fund recipients must use evidence-based 
approaches (Tiers I-IV) when evidence “is reasonably available,” 
as determined by states. Some ESSA programs provide compet- 
itive preference for implementing evidence-based interventions 
that meet Tiers I-Ill. Because of these different requirements and 
incentives, it is important for decision-makers and afterschool 
providers to understand which programs authorized under ESSA 
can support afterschool activities and whether their states have 
set a minimum evidence level for these programs. Evidence 


requirements for relevant programs are summarized in Box 3. 


e There is at least one statistically significant improved 
outcome, but there are more “no effect” (not statistically 
significant) outcomes than improved outcomes. 


No Effect. In this review, an outcome domain is described as 
demonstrating no effect when: 


- There are no statistically significant improved outcomes 
or statistically significant negative outcomes. 


Negative Effect. In this review, an outcome domain is described 
as demonstrating a negative effect when: 


- There is at least one statistically significant negative 
outcome and no statistically significant improved outcomes, 
OR 


- There is at least one statistically significant negative 
outcome and at least one statistically significant 
improved outcome, but there are more negative than 
improved outcomes. 


BOX 3 


ESSA programs supporting afterschool activities and 


their evidence requirements 


ESSA formula grant activities requiring evidence 
of effectiveness 


Most of Title I, Part A funds are distributed to states through 

a federal formula and can be used for afterschool activities 
consistent with the purposes of the Title. Under ESSA Title I, Part 
A, Section 1003, states must set aside 7 percent of those funds 
to help support school improvement plans, which ESSA requires 
for schools designated by states as needing improvement. 

Each school improvement plan must include evidence-based 
interventions (i.e., Tiers I-IV). For those schools that receive funds 
from the 7 percent set-aside for low-performing schools, at least 
one intervention in each school’s improvement plan must be 
supported by evidence at Tiers I-Ill. 


ESSA formula grant activities that encourage evidence 
of effectiveness 


21st Century Community Learning Centers (Title IV, Part B). The 
primary federal source for afterschool funding, the 21st Century 
Community Learning Centers program, is authorized under Title 
IV, Part B of ESSA. Funding for this program is distributed to 
states through a formula. States then use the bulk of this to fund 
afterschool providers through a competitive process. Providers 
can be local education agencies, community organizations, or 
other public or private entities. Authorized activities include 
academic enrichment, including tutoring, and a broad array of 
other services and activities, such as arts, music, health, social 
and emotional development, physical fitness, nutrition, substance 
abuse prevention, career and technical programs, and internship 
or apprenticeship programs, among others. 


Direct Student Services funded under Title | (Section 1003A). 
ESSA allows states to set aside up to 3 percent of Title | funds 
for Direct Student Services, including “high-quality academic 
tutoring” that meets criteria outlined in ESSA (Section 1003(e)). 
Under No Child Left Behind (NCLB), a previous authorization 
of the law, these services were known as Supplemental 
Educational Services (SES); Title | schools that had not made 
Adequate Yearly Progress for three years were required to 
offer this option to their low-income students. Some of these 
SES programs fit our definition of an afterschool program, and 
several studies of SES are included in this review. ESSA made 


such services optional, and states may elect to require that 
program models have evidence of effectiveness particularly as 
part of examining their demonstrated record of success during 
the state approval process for providers. 


Title I, Part A formula grants for Targeted Assistance Programs 
(Section 1115). In targeted assistance schools, Title | funds may 
be used only to provide supplemental instructional services to 
students identified as having the greatest need for additional 
help. Such services may include out-of-school time and expanded 
learning time programs, including programs held before or after 
school and during the summer. 


Federal discretionary grants that require evidence 
of effectiveness 


ESSA authorizes seven discretionary grant programs that 
award competitive points for evidence of effectiveness, 
per ESSA requirements. Of these, four could be used for 
afterschool programming. 


Promise Neighborhoods (Section 4624(b)). This program 
supports neighborhood-based, cross-sector, “cradle-to-career” 
solutions to supporting academic success. 


Full-Service Community Schools (Section 4625(b)(3)). This 
program supports partnerships between education agencies 
and community-based and nonprofit organizations to provide 
comprehensive services for students and families. 


Jacob K. Javitz Gifted and Talented Students Education 
Program (Section 4644(f)(2)). A key purpose of the program is 
to support programs and projects for identifying and serving 
gifted and talented students, including through out-of-school 
time programs. 


Literacy for All, Results for the Nation (LEARN) grants (ESSA, 
Section 2224(b)). These grants to states support development 
of comprehensive literacy plans and a range of other activities, 
including professional development for educators and 
stakeholder engagement. Funds unexpended for these key 
activities may be used to support services for students, including 
“connecting out-of-school learning opportunities to in-school 
learning to improve children’s literacy achievement.” 


Understanding the Program Summary overview page 


ACADEMIC PROGRAMS 


Programs with at least one study meeting ESSA Tier I-Ill research 21st Century Community Learning Centers 
: Zoe . (Pittsburgh Charter Schools - Elementary) 
requirements for establishing cause-and-effect have a detailed 


summary of findings. Programs with multiple studies have 


summaries for each study, as well as an overall summary of Eidabaenhs 2st Coty Cowman Lestnina Coates Ses 

aimed to improve academic and behavioral outcomes of LOCALE REGION 
low-income youth. This program, offered in two charter schools 

in Pittsburgh during 2008-09, provided homework help and j 


findings across studies. 


specialized instruction in areas of academic weakness. The 


program was offered for the full school year, five days per 


The first page of each summary provides an overview of the week, for 2.5 hours each day. 
Urban Northeast 
program and the study, including: 
. F feat Participants 
¢ A brief narrative description of the program, CLASS GRADE TARGET STUDENTS 
| Ges mtn 
« An Effectiveness-At-A-Glance summary of the program’s 
Deco a SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


effects in any of 10 outcome domains studied, and 


The study found at least one improved outcome with no 
overriding contrary evidence for 


« Information-in-brief about the program location and Mat erste caveat 
Reading/ELA Achievement 


participants, program features, and implementation. 


NOT MEASURED FOR THIS INTERVENTION: General Achieve 


84% Low-Income 74% Black 
More detail on each element of the page is provided below. Giaslusibe Atarantice A Enfoliqaft School Enaaaeman eetenes 13% White 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework support SETTING: School site 
BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff 
DURATION: Full school year, 5 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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The Effectiveness-At-A-Glance summary provides a narrative 
EFFECTIVENESS AT A GLANCE 


description of the overall effectiveness of the program. 
Effectiveness is summarized by outcome domain (groupings of 


related outcome measures).' This summary addresses the U.S. Positive Effect 


Department of Education’s recommendation that decision-makers Tie stiniy iguana lecscone imorovediouieome wihinc 


examine the full body of evidence about a program. overriding contrary evidence for 
Each domain studied receives one of four descriptors: (1) Positive Mathematics Achievement 
Effect; (2) Mixed Effects; (3) No Effect; or (4) Negative Effect (see Reading/ELA Achievement 


Box 2 for more detail about these descriptors). 


It is important to note that it is possible for programs to have 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


evidence of a statistically significant improved outcome but ment, Other Achievement, Physical Activity/Health, Promotion and 
Graduation, Attendance & Enrollment, School Engagement, Science 


not have a Positive Effect rating in that outcome domain. One Achievement and Social & Enorlonal eon petencies 


instance when this would occur is when another outcome in that 
domain shows a negative effect. 


1 This review examined ten outcome domains: Attendance & Enrollment; Engagement with 


Schooling; General Achievement; Mathematics Achievement; Physical Activity/Health; Promotion 
& Graduation; Reading/English Language Arts (ELA) Achievement; Science Achievement; Social & 


Emotional Competencies; and Other Achievement. 
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Study Location and Participants. This section presents 
information reported by study authors about where the study 
took place and the characteristics of students who participated 
in the study. We also indicate when the authors do not provide 
this information. 


« Locale can be urban, suburban, or rural, or combinations 
of these. 


+ Region can be West, Midwest, Southwest, Southeast, or 
Northeast. We identified no studies conducted in Alaska 
or Hawaii. 


* Class Grade is the grade in school for participants as 
reported in the study. Some authors report student age 
rather than grade. 


+ Target students identifies any specific types of students 
to which a program was targeted. For example, some 
programs are intended for overweight students or for 
students with academic challenges. Programs that 
are not specifically targeted are identified as general 
population programs. 


+ Socio-economic status is the percentage of students in 
the study eligible for free- or reduced-price lunch, unless 
otherwise indicated. 


+ Race or ethnicity summarizes the distribution of race 
or ethnicity of study participants. For some studies, 
reporting is incomplete, and percentages do not sum to 
100 percent. 
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STUDY LOCATION AND PARTICIPANTS 


Urban Northeast 


Participants 


CLASS GRADE TARGET STUDENTS 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


84% Low-Income 74% Black 
13% White 
13% Other 
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Program Content. This section provides basic information about 
program components and practices. Study authors reported this 


Program Content Logistics 


information inconsistently, and we provide only the information MAIN. COMPONENTS: Thtarina,homewerlesubport School sie 


BEST PRACTICES USED: Sequenced, active learning Group and one-to-one formats 


we were able to gather from the studies. Further, although 


reviewers were trained to identify certain program features, there 
was necessarily some subjective judgment. We indicate when the 
authors do not provide this information. 


«+ Main components. These are the primary program activities, 
as described by the study. Commonly-reported components Program Content 
include tutoring, homework help, sports, drama, music, MAIN COMPONENTS: Tutoring, homework support 


crafts, apprenticeships, and mentoring. 


BEST PRACTICES USED: Sequenced, active learning 


+ Best afterschool practices used. We looked for indications 
of five types of practices that can contribute to strong 


afterschool programming: 


« Sequenced programming. Programs that use a connected 
and coordinated set of activities to achieve objectives 
offer sequenced programming. One indication of 
sequenced programming is a program manual or set 
of lesson plans. 


* Active learning. We looked for indications that the 
program used active forms of learning to help youth 
learn new skills. 


- Positive adult-student relationships. We identified a 
program as having positive adult-student relationships 
when the study explicitly mentioned any of the following: 
youth have at least one highly supportive relationship 
with an adult at the program, the program is intentional 
about developing youth/adult relationships (e.g., builds 
in one-on-one time between youth and adult), there is a 
low child-to-staff ratio (10:1 or less), or low staff turnover. 


« Family engagement. We identified a program as having 
family engagement when the study explicitly mentioned 
any of the following: program staff are skilled or trained 
in working with youth and families, program staff are 
familiar with or training in cultural/language barriers that 
exist or families of program students, or families receive 
communication in their own language through various 
methods (flyers, email, phone, face-to-face) when 
possible regarding the child’s needs and progress. 


* Cultural elements. We identified a program as having 
cultural elements when the study explicitly mentioned 
any of the following: program activities or materials that 
encourage cultural exploration and heritage, program 
has opportunities for participants to interact with youth 
who have different cultural identities, or program 
collects demographic data on participants and seeks to 
identify underserved populations. 
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Program Logistics. This section provides information on program 
implementation features. This information can help decision- 
makers interpret the findings about effectiveness (for example, 
was the program likely intensive enough to produce an effect 

on an outcome?). This information can also to rule in or rule out 


programs depending on the resources required. 


Study authors reported this information inconsistently, and we 
provide only the information we were able to gather from the 
studies. Further, although reviewers were trained to identify 
certain program features, there was necessarily some subjective 
judgment. We indicate when the authors do not provide this 
information. Further, we note that program cost was almost never 
reported by study authors; for this reason, we do not report cost. 


¢ Setting can be school site(s), community site(s), or both. 


- Delivery format can be group setting, one-to-one 
formats (such as tutoring or mentoring), or both. 


* Staffing can include certified teachers, program staff, 
volunteers, and/or specialized staffing. Specialized 
staffing may include therapists or others with special 
skills, such as university professors or scientists. 


¢ Duration indicates how often the program was offered 
and for how long. When information is available about 
how often students actually attended the program 
(for example, average number of days attended), we 
include it. 


¢ Implementation quality. We looked for four indicators 
of implementation quality that can contribute to strong 
afterschool programming: (1) availability of a program 
manual; (2) provider training for implementing this 
program; (3) ongoing support for implementation 
through supervision, consultation, coaching, booster 
sessions, debriefing, or other forms of support for the 
program providers; or (4) evidence that at least some 
of the program staff had cultural backgrounds similar 
to those of major groups of students participating in 
the program. 
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Program Content Logistics 
SETTING: School site 


Sequenced, active learning DELIVERY FORMAT: Group and one-to-one formats 
STAFFING: Program staff 
DURATION: Full school year, 5 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 


J 


Logistics 


SETTING: School site 

DELIVERY FORMAT: Group and one-to-one formats 
STAFFING: Program staff 

DURATION: Full school year, 5 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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i ocak a | smccsmsrncremem 
The Study Details page(s) for each program will be of greatest interest to those Ue ey See 


who want to dig deeper into the research itself. This page provides information Study Details 


STUDY DESCRIPTION 


about the study design, the types of outcomes that were studied, and the 
specific measures used. Each outcome is characterized by the ESSA evidence 
tier it meets under different circumstances (described in more detail below), 


the group of students for whom the outcome was measured, the number of 
students, and the size and statistical significance of the effect. 


Tier il Tier It Grades 3-5 55 0.60 (0.28)" 


‘Average effect size for Mathematics Achievement outcomes at Tiers lll 0.60 


- Evidence tier. A key piece of information is the ESSA evidence tier 


that each outcome could meet. We present the ESSA tier for two pecbbieeriaana es 
conditions: (1) without site and sample size criteria and (2) with site and 


sample size criteria. The U.S. Department of Education recommends 


Tier ll Tier it Grades 3-5 56 0.60 (0.28)" 


that Tier I-Il evidence come from a supporting study (or studies) that faery eterno ume wea een ll NE 


involve more than one site (school district) and have a sample size 

of at least 350. In the afterschool context, it is common for studies that 
are Tier | or Il when sample and site criteria are not applied to fall to 
Tier Ill when these criteria are used. 


¢ Group. This provides information about the grades, cohorts, _ evoneccurcronsrnnciccmesis 18 
or other groupings that were measured in some studies. This 
information is especially relevant when effects for students in 
different grades or program years were assessed separately. 


« Effect size. This is a common way to measure the size of the program’s 
effect for an outcome. There is no hard-and-fast rule for determining 
whether an effect size is small, medium, or large, and typical effect 
sizes vary by type of outcome and age of student. We advise that 
decision-makers use effect sizes judiciously, being sure to compare 
effect sizes only for similar outcomes and similar types of students. 


¢ Statistical significance. Outcomes with statistically significant 
differences (p<.05), either positive or negative, between program 
and comparison groups are identified with an asterisk. 
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Quick-Links to 


Program Summaries 


These links enable quick navigation within the electronic document to programs of interest. 
The links are organized by afterschool program type. Within each program type, programs with 


at least one statistically significant positive outcome are listed first, by grade level. 


Academic programs 2 


In these programs, students may receive tutoring in academic content areas and help with homework. 
Although the program may include other activities such as recreation and enrichment, most of the time is 


spent on academic activities. 


ACADEMIC PROGRAMS WITH AT LEAST ONE STATISTICALLY SIGNIFICANT IMPROVED OUTCOME 


For elementary students 


21st Century Community Learning Centers (Pittsburgh charter schools - Elementary) 


Academically-focused extended day program 


Afterschool reading tutoring (Texas 


Afterschool tutoring program 


Building Educated Leaders for Life (BELL) 


ELA Extra and Math Mania 


Raising Education Attainment Challenge 


For middle grades students 


Middle School Academic Intervention Program 


Supplemental Educational Services (Los Angeles — Middle Grades) 


Warrior After School 


For elementary and middle grades students 


Supplemental Educational Services (Chicago) 


Supplemental Educational Services (Dallas) 


Supplemental Educational Services (Florida) 


Supplemental Educational Services (Minneapolis) 


For high school students 


Academic Volunteer Mentor Service Program 


Supplemental Educational Services (Los Angeles — High School) 
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ACADEMIC PROGRAMS WITH NO EVIDENCE OF IMPROVED OUTCOMES 


For elementary students 


* 21st Century Community Learning Centers (Georgia — Elementary) 


« Afterschool literacy program with Barton Reading & Spelling System® 


¢ LeapTrack 
+ Math Academic Enhancement Program 


« Math program with ST Math 


+ Purple Sage Elementary Writing Project 


+ Supplemental Educational Services (Los Angeles — Elementary) 


¢ William Penn Initiative for Neighborhood Success 


For middle grades students 


* 21st Century Community Learning Centers (Georgia —Middle Grades) 


+ 21st Century Community Learning Centers (Pittsburgh charter schools - Middle Grades) 


« Academic Intervention Services (New York) 


« Academic Remediation Program (South Carolina) 


- After-School Academy 


¢ Intervention Services Inc. tutoring program 


« Language Workshop 


- New Jersey test prep tutoring program 


For elementary and middle grades students 


- Academic Achievement Academy — math tutoring 


« Academic Achievement Academy — reading tutoring 


*« Supplemental Educational Services (Louisville) 


¢ Supplemental Educational Services (Milwaukee) 


« Supplemental Educational Services (Mississippi) 


¢ Supplemental Educational Services (Virginia) 


For high school students 
« College Possible 


« Texas afterschool math remediation 


« Supplemental education program (Texas) 


Grade levels not reported 


« Supplemental Educational Services (New Mexico) 


Afterschool plus social supports 2 


These programs incorporate case management and more extensive family involvement in 
addition to a multicomponent program that includes a range of activities. 


AFTERSCHOOL PLUS SOCIAL SUPPORTS PROGRAMS WITH NO EVIDENCE OF 
IMPROVED OUTCOMES 


For middle grades students 


« Challenging Horizons 


For high school students 
* Quantum Opportunity 
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Arts 


These programs primarily focus on introducing students to one or more art forms 
(for example, visual arts, dance, theater, music) and/or developing their artistic skill. 


ARTS PROGRAMS WITH NO EVIDENCE OF IMPROVED OUTCOMES 


For middle grades students 


« Theater arts afterschool program 


Career and leadership 7 


These programs are a type of Multicomponent program that focuses on career development, 
postsecondary readiness, and/or leadership development. 


CAREER AND LEADERSHIP PROGRAMS WITH AT LEAST ONE STATISTICALLY 
SIGNIFICANT IMPROVED OUTCOME 


For middle grades students 
+ Citizen Schools 


« Stay-in-School for College and Career Opportunities (SISCO) 


For high school students 
« After School Matters 


« Hispanic Youth Leadership Program 


CAREER AND LEADERSHIP PROGRAMS WITH NO EVIDENCE OF IMPROVED OUTCOMES 


For middle grades students 
* LEAD Academy 


« Time Travelers Leadership Program 


For high school students 


« Ellas/Ellos Mentoring Program 


Multicomponent 7 


Multicomponent programs offer multiple types of activities, and no type of activity dominates 
the time youth spend in the program. Multicomponent programs typically offer academic support, 
sports, and arts and crafts programming, and youth can experience multiple of these components. 


MULTICOMPONENT PROGRAMS WITH AT LEAST ONE STATISTICALLY 
SIGNIFICANT IMPROVED OUTCOME 


For elementary students 


+ 21st Century Community Learning Centers (Fresno) 
* 21st Century Community Learning Centers (Philadelphia - Elementary) 


« After School Education and Safety (California — Elementary) 


¢ Baltimore Community Schools (Elementary) 


« LA’s Better Educated Students for Tomorrow (LA’s BEST) 
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« Multicomponent program (Northeast) 


¢ Tutoring and enrichment program 


For middle grades students 


* 21st Century Community Learning Centers (Marietta Boys and Girls Club) 


* 21st Century Community Learning Centers (Middle Grades) 


* 21st Century Community Learning Centers (Philadelphia — Middle Grades) 


« After School Education and Safety (California — Middle Grades) 


¢ The After-School Corporation (Middle Grades) 


« AfterZone 


¢ Baltimore Community Schools (Middle Grades) 


« Cooke Middle School Afterschool Recreation Program 


« Texas After School Initiative 


For elementary and middle grades students 


* 21st Century Community Learning Centers (St. Louis, Missouri) 


* The After-School Corporation (PreK — Grade 8) 


« Cool Girls 


For high school students 
* 21st Century Community Learning Centers (Philadelphia — High School) 


- After School Safety and Enrichment for Teens 


* The After-School Corporation (High School) 


MULTICOMPONENT PROGRAMS WITH NO EVIDENCE OF IMPROVED OUTCOMES 


For elementary students 


* 21st Century Community Learning Centers (Elementary) 


* 21st Century Community Learning Centers (Louisiana) 


* 21st Century Community Learning Centers (New Hampshire) 


« 21st Century Community Learning Centers (West Virginia - Elementary) 


« After School Education and Safety (San Francisco — Elementary) 


« Boston Elementary School After-School Program 


« Hope Center for Kids 


For middle grades students 


* 21st Century Community Learning Centers (West Virginia — Middle Grades) 


« After School Education and Safety (San Francisco — Middle Grades) 


« Afterschool Program with All Stars Prevention Curriculum 


« Santa Ana After School Learning and Safe Neighborhoods Partnerships Program 


« School-to-Jobs 


¢ Support Our Students 


For high school students 
*« 21st Century Community Learning Centers (West Virginia — High School) 


¢ Baltimore Community Schools (High School) 
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Physical activity/health 7 


In these programs, most of the time is focused on healthy living and physical activity. 


PHYSICAL ACTIVITY/HEALTH PROGRAMS WITH AT LEAST ONE STATISTICALLY 
SIGNIFICANT IMPROVED OUTCOME 
For elementary students 

« America SCORES 

- FITKids 

+ Georgia Prevention Institute Physical Activity Program 


« Girls inthe Game 
¢ LA Sprouts 
« Youth Fit for Life 


For middle grades students 


« Fitness-focused afterschool programs (California) 


« Girls on the Move 


+ Student-centered physical activity program 


PHYSICAL ACTIVITY/HEALTH PROGRAMS WITH NO EVIDENCE OF IMPROVED OUTCOMES 


For elementary students 


« Afterschool exercise program 


« Scouting Nutrition & Activity Program 
« GEMS 


¢ Girlfriends for KEEPS 


« Physical activity program for boys 


« SPARK program 


For middle grades students 


« Afterschool physical activity program 


For high school students 
« COPE TEEN 


Science, Technology, Engineering, and Mathematics (STEM) 2 


In STEM programs, participants develop their interests in science, technology, engineering, and/or 
mathematics-related topics. Improving school academic performance is generally not the focus of these 
programs; instead, the programs seek to engage students in hands-on, active activities that nurture an 
interest in STEM fields. 


STEM PROGRAMS WITH AT LEAST ONE STATISTICALLY SIGNIFICANT IMPROVED OUTCOME 


For elementary students 

« 4-H Robotics Program 

+ Bringing Up Girls in Science (BUGS) 
For middle grades students 


- The Investigators Club 


For high school students 
+ FIRST Robotics 
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STEM PROGRAMS WITH NO EVIDENCE OF IMPROVED OUTCOMES 


For elementary students 


* FIRST Lego League 


For middle grades students 


« Leonardo’s Laboratory 


« STEM Excellence and Leadership 


Sports 2 


Sports programs have a competitive component. Examples include running clubs and school-sponsored 


team sports. 


SPORTS PROGRAMS WITH AT LEAST ONE STATISTICALLY SIGNIFICANT IMPROVED OUTCOME 


For middle grades students 


+ School-sponsored sports (Texas) 


For high school students 


« High school interscholastic sports 


« School-sponsored sports (Miami) 


« School-sponsored sports (South Texas) 


SPORTS PROGRAMS WITH NO EVIDENCE OF IMPROVED OUTCOMES 


For middle grades students 


« SquashSmarts 


For high school students 


« School-sponsored sports (Career & technical school) 
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SUMMARIES 


Academic Programs 


ACADEMIC PROGRAMS 


PARAS iigaGonviinting Mer lentim aeolian 


(Pittsburgh Charter Schools - Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Pittsburgh’s 21st Century Community Learning Centers Location 


aimed to improve academic and behavioral outcomes of LOCALE REGION 
low-income youth. This program, offered in two charter schools 

in Pittsburgh during 2008-09, provided homework help and 

specialized instruction in areas of academic weakness. The 

program was offered for the full school year, five days per i | 


week, for 2.5 hours each day. 


Urban Northeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 3-5 General Population 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


; : a 5 84% Low-Income 74% Black 
ment, Other Achievement, Physical Activity/Health, Promotion and . 
Graduation, Attendance & Enrollment, School Engagement, Science 13% White 
Achievement, and Social & Emotional Competencies 13% Other 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework support SETTING: School site 
BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff 
DURATION: Full school year, 5 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (PITTSBURGH CHARTER SCHOOLS - ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool tutoring and homework help program in two charter schools in Pittsburgh, 
Pennsylvania. The treatment group for mathematics is defined as students who attended at least 50 percent of the days when the 
program focus was mathematics. The treatment group for reading is defined as students who attended at least 50 percent of the days 
when the program focus was reading. The comparison students experienced “business as usual,” which could have included other 
school- and community-sponsored afterschool activities. The study was conducted in 2008-09. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome 5 ‘ ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


A4Sight Pennsylvania 
Core Standards 
Benchmark Tier Ill Tier Ill Grades 3-5 55 0.60 (0.28)* 


Assessments - 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill | oso 


mathematics 
* Statistically significant at p<.05 
READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : i ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
ASight Pennsylvania 
Core Standards 3 
Tier Ill Tier Ill Grades 3-5 56 0.60 (0.28)* 


Benchmark 
Assessments - reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III | oso 


* Statistically significant at p<.05 


CITATION: Dreyer, K. J. (2010). An examination of academic outcomes for students who attend a school-based afterschool program (Doctoral disserta- 
tion, University of Pittsburgh). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3417442) 
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ACADEMIC PROGRAMS 


Academically Focused Extended Day Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This academically focused extended day program, offered in Location 
one New York City elementary school during 2001-02, aimed to LOCALE REGION 
improve student academic performance with an emphasis on 

math computation and basic reading skills. Students completed 

sequenced sets of multi-item worksheets, matched to their 

skill levels. The program was offered for the full school year, i | 

three days per week, for one hour each day. The program was 


staffed by teachers from the host school. 


Urban Northeast 
EFFECTIVENESS AT A GLANCE 
Participants 
CLASS GRADE TARGET STUDENTS 
Positive Effect 
PreK - Grade 5 Low-Income 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mixed Effects 


The study found a mix of improved and null or negative 
outcomes for 


Reading/ELA Achievement 


OUTCOMES THAT DO NOT MEET TIER I-IIl STANDARDS FOR 
CAUSALITY: Social & Emotional Competencies 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 


100% Low-Income 66% Black 
17% Latino 


Graduation, Attendance & Enrollment, School Engagement, and 17% Other 
Science Achievement 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Individualized, one-to-one format 


STAFFING: Certified teachers 
DURATION: Full school year, 3 days/week, 1 hour/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | ACADEMICALLY FOCUSED EXTENDED DAY PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool supplemental instruction program in one New York City elementary 
school. Outcomes for students who participated were compared to outcomes for a matched group of non-participants. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 
The study was conducted in 2001-02. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ¥ ar Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Evaluator-created 
P P ‘ ‘ Grades * 
formative mathematics Tier Ill Tier Ill Prek-1 94 0.60 (0.25) 
assessment 
Evaluator-created 
é , F : Grades * 
summative mathematics Tier Ill Tier Ill Prek-l 94 0.79 (0.27) 
assessment 
Evaluator-created 
formative mathematics 8 8 Grades 4-5 62 -0.42 (0.26) 
assessment 
Evaluator-created 
summative mathematics Tier Ill Tier Ill Grades 4-5 62 0.61 (0.26)* 


assessment 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


* Statistically significant at p<.05 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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ACADEMIC PROGRAMS | ACADEMICALLY FOCUSED EXTENDED DAY PROGRAM 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome é “ tags Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Evaluator-created Grad 
formative reading § 8 Bice 94 0.44 (0.25) 
assessment 
Evaluator-created Grad 
summative reading Tier III Tier III Bralet 94 0.54 (0.26)" 
assessment 
Evaluator-created 
formative reading 8 8 Grades 4-5 62 0.03 (0.26) 
assessment 
Evaluator-created 
summative reading 8 8 Grades 4-5 62 0.09 (0.26) 
assessment 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.27 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


CITATION: Chatterji, M., Kwon, Y. A., & Sng, C. (2006). Gathering evidence on an afterschool supplemental instruction program: Design challenges and 
early findings in light of NCLB. Education Policy Analysis Archives, 14(12), 1-47. 
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ACADEMIC PROGRAMS 


Afterschool Reading Tutoring (Texas) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This afterschool reading tutoring program, offered from Location 
2006-07 to 2009-10 in one Texas school, aimed to improve LOCALE REGION 
the reading achievement of low-income third grade students 

who did not meet state reading standards. Certified teachers Fy 


offered small-group tutoring using activities drawn from the 


school day curriculum, as well as other resources. Tutoring | 
activities were based on objectives for which students had not 
et demonstrated mastery. The duration of the program was 
y y | Suburban, Urban Southwest 
not reported. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grade 3 Low-Income 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical 


tas! i A 90% Low-Income 57% Latino 1% Asian 
Activity/Health, Promotion & Graduation, Attendance & Enrollment, 
School Engagement, Science Achievement, and Social & Emotional 35% Black 1% Other 
Competencies 8% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | AFTERSCHOOL READING TUTORING (TEXAS) 


Study Details 


STUDY DESCRIPTION 

In this quasi-experiment, outcomes for third grade students who participated in the afterschool tutoring program were compared to out- 
comes for a matched group of non-participants. The comparison students experienced “business as usual,” which could have included 
other school- and community-sponsored afterschool activities. The study was conducted in one school in Texas in 2006-07 to 2009-10. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; oe Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 
Texas Assessment 
of Knowledge and Tier Ill Tier Ill Grade 3 152 0.72 (0.17)* 

Skills - reading 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.72 


* Statistically significant at p<.05 


CITATION: Bridges, C. (2011). The correlation between after school tutoring and reading scores of at-risk third graders (Doctoral dissertation, Walden 
University). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3482150) 
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ACADEMIC PROGRAMS 


Afterschool Tutoring Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This afterschool tutoring program, offered in one school in a Location 
medium-sized Midwestern city in 1998-99, aimed to improve LOCALE REGION 
the academic performance of low-achieving fourth grade stu- 
dents through afterschool small-group tutoring and homework 
help. The program was staffed by a teacher and volunteers and 
offered for the full school year, six hours per week. i | 

Urban Midwest 
EFFECTIVENESS AT A GLANCE 
Positive Effect Participants 
The study found at least one improved outcome with no irnae GRADE Thee eee 
overriding contrary evidence for 

Grade 4 Low-Achieving Students 


General Achievement 
Mathematics Achievement 


. ; SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Reading/ELA Achievement 


No Effect 


The study found neither improved nor negative outcomes for 
Other Achievement 


Social & Emotional Competencies 


: Not Reported 21% White 
NOT MEASURED FOR THIS INTERVENTION: Physical 
Activity/Health, Promotion & Graduation, School Engagement, 79% Other 
and Science Achievement 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help SETTING: School site 
BEST PRACTICES USED: Positive adult- DELIVERY FORMAT: Group format 


lationshi 
SAE rhe evenetNRe STAFFING: Program staff, volunteers 


DURATION: Full school year, 6 hours/week 


IMPLEMENTATION SUPPORT: None reported 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 30 


ACADEMIC PROGRAMS | AFTERSCHOOL TUTORING PROGRAM 


Study Details 


STUDY DESCRIPTION 


This quasi-experiment estimates the impact of an afterschool tutoring program for fourth graders. Outcomes for students who partici- 
pated were compared to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” 
which could have included other school- and community-sponsored afterschool activities. The study was conducted in one school a 


medium-size Midwestern city in 1998-99. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Outcome a e a Group 
Without site and With site and 
sample criteria sample criteria 


Grade point average 


— 2 quarters into 8 8 Grade 4 
intervention 
Grade point average — 4 5 
x : Tier Ill Tier Ill Grade 4 


end of 1 year 


Average effect size for General Achievement outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


Number of Effect size 

students (Standard error) 
Cannot 

oP determine 
Cannot 

3 determine* 


Cannot 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Outcome . ¥ ay. Group 
Without site and With site and 
sample criteria sample criteria 


Mathematics course 


grade — 2 quarters into Tier Ill Tier Ill Grade 4 
intervention 
Mathematics course Z 2 
Tier Ill Tier Ill Grade 4 


grade — end of 1 year 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


Number of Effect size 

students (Standard error) 
Cannot 

wy determine* 
Cannot 

78 determine* 


Cannot 


determine 
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ACADEMIC PROGRAMS | AFTERSCHOOL TUTORING PROGRAM 


OTHER ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ri . ot Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Handwriting grade — 
2 quarters into 8 § Grade 4 66 Cannot 


: : determine 
intervention 

Handwriting grade — Cannot 
end of 1 year 8 8 mide a determine 


é : : Cannot 
Average effect size for Other Achievement outcomes at Tiers I-IIl 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome E ; ner Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
English language 
arts course grade — Le 7 Cannot 
dl Tier Ill Tier Ill Grade 4 77 : 

2 quarters into determine* 


intervention 


English language 


arts course grade — Tier Ill Tier Il Grade 4 75 Pee 
end of 1 year 


Reading course Cc 
grade — 2 quarters 8 8 Grade 4 77 eAnot 


into intervention detetmilic 
peal peed - § § Gidea 75 eect 
Mee ine 8 8 Giade# i Fae 
ne ae a g g Grade 4 74 ao e 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III earner 


determine 


* Statistically significant at p<.05 


§ Meets Tier III standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 32 


ACADEMIC PROGRAMS | AFTERSCHOOL TUTORING PROGRAM 


SOCIAL & EMOTIONAL COMPETENCIES 


ESSA evidence tier 
Number of Effect size 
Outcome , ; Tae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Conduct grade — 
2 quarters into Cannot 
8 8 Grade 4 77 determine 
intervention 
Conduct grade — Cannot 
end of 1 year 8 8 Graded I determine 
Effort grade — c : 
5 anno 
2 quarters int : 
quarters i 8 8 Grade 4 77 determine 
intervention 
Effort grade — Cannot 
r 4 7 : 
end of 1 year 8 S Grade 5 determine 


: : . : , Cannot 
Average effect size for Social & Emotional Competencies outcomes at Tiers I-Ill 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Zosky, D. L., & Crawford, L. A. (2003). No Child Left Behind: An assessment of an after-school program on academic performance among 
low-income at-risk students. Schoo! Social Work Journal, 27, 18-31. 
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ACADEMIC PROGRAMS 


Building Educated Leaders for Life (BELL) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Building Educated Leaders for Life (BELL) aimed to improve Location 


the reading and math achievement of low-income students LOCALE REGION 
in grades K-6 through academic support and enrichment. 

Certified teachers offered small-group instruction in reading 

and math using a literature-based reading curriculum (Voices 

of Love and Freedom) and Houghton Mifflin’s MathSteps i | 

curriculum. This program, held in seven New York City schools 

during 2004-05, was offered for most of the school year (24 


Urban Northeast 
weeks), three days per week, for 1.5 hours each day. 


EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 


Positive Effect Grades 3-6 Low-Income 


The study found at least one improved outcome with no 


overriding contrary evidence for 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mathematics Achievement 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


: : im : 72% Low-Income 49% Black 1% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & : 
Graduation, Attendance & Enrollment, School Engagement, Science 48% Latino 1% Other 
Achievement, and Social & Emotional Competencies 1% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Positive adult-student DELIVERY FORMAT: Group format 


relationships, respect for student cultures a 
ip 2 _ pee STAFFING: Certified teachers, volunteers, program staff 


DURATION: 24 weeks, 3 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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ACADEMIC PROGRAMS | BUILDING EDUCATED LEADERS FOR LIFE (BELL) 


Study Details 


STUDY DESCRIPTION 

This quasi-experimental study was conducted in seven New York City public schools served by Building Educated Leaders for Life 
(BELL) during 2004-05. Outcomes for students who participated were compared to outcomes for a matched group of non-participants. 
The comparison students experienced “business as usual,” which could have included other school- and community-sponsored after- 
school activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome 5 e a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


CTB McGraw-Hill 
assessment: Tier Ill Tier Ill Grades 3-6 690 0.23 (0.08)* 
mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . % ar Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


CTB McGraw-Hill 
assessment: English 8 8 Grades 3-6 690 -0.06 (0.08) 


language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Cooper, T. M. (2007). The effects of supplemental educational services on student achievement (Doctoral dissertation, Boston College). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3262811) 
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ACADEMIC PROGRAMS 


ELA Extra and Math Mania 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The ELA Extra and Math Mania afterschool programs, offered Location 


in one school district in New York state during 2004-05, aimed LOCALE REGION 
to improve the literacy and math skills of suburban fourth 

grade students who scored below the 50th percentile on third 

grade standardized tests. Certified teachers used the Best 

Practices in Reading program and the Foundations in Math 4 

curriculum. The program was offered for 22 weeks, three days 


per week, for one hour each day. ee Nopthease 


St od Oe AYA We cd NV) ea as 
Participants 


CLASS GRADE TARGET STUDENTS 


Positive Effect 
Grade 4 Low-Achieving Students 
The study found at least one improved outcome with no 


overriding contrary evidence for 


Mathematics Achievement SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0 4 9 j 
ment, Other Achievement, Physical Activity/Health, Promotion and 57% Low-Income 56% Latino 
Graduation, Attendance & Enrollment, School Engagement, Science 22% Black 
Achievement, and Social & Emotional Competencies 22% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: 22 weeks, 3 days/week, 1 hour/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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ACADEMIC PROGRAMS | ELA EXTRA AND MATH MANIA 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on mathematics and English language arts achievement of ELA Extra and Math Mania 

for fourth graders in one school district in New York state. Outcomes for students who participated were compared to outcomes for a 
matched group of non-participants. The comparison students experienced “business as usual,” which could have included other school- 
and community-sponsored afterschool activities. The study was conducted in 2004-05. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome a : a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
New York State Testin 2 F 
g Tier Ill Tier Ill Grade 4 188 0.42 (0.15)* 


Program - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : eer Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
New York State Testin 
2 § § Grade 4 188 0.22 (0.5) 


Program - language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Gill, M. J. (2006). Effective literacy and math instruction, better student achievement: An evaluative study of afterschool programming 
(Doctoral dissertation, Fordham University). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3210266) 
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ACADEMIC PROGRAMS 


Raising Education Attainment Challenge (REACH) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Raising Educational Attainment Challenge (REACH) Location 
afterschool tutoring program aimed to increase the reading LOCALE REGION 
skills and standardized test scores of low-achieving third and 

fourth grade students at an urban elementary school in the 

Midwest. Certified teachers provided homework help and 

offered individual literacy tutoring using high-interest material. i | 

The program was offered from January through April, five days 


per week, for up to one hour per day. 


Urban Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 3-4 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


mene Mathematics penlevement, Other Achievement, Physical 100% Low-Income 52% Black 
Activity/Health, Promotion & Graduation, Attendance & Enrollment, 

School Engagement, Science Achievement, and Social & Emotional 27% White 
Competencies 20% Latino 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Individualized, one-to-one format 


STAFFING: Certified teachers 
DURATION: January-April, 5 days/week, 30-60 minutes/day 


IMPLEMENTATION SUPPORT: Program manual 
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ACADEMIC PROGRAMS | RAISING EDUCATION ATTAINMENT CHALLENGE (REACH) 


Study Details 


STUDY DESCRIPTION 

This study uses a quasi-experimental approach to estimate the impact of the Raising Education Attainment Challenge (REACH) 
afterschool tutoring program. Outcomes for students who participated were compared to outcomes for a matched group of non- 
participants. The comparison students experienced “business as usual,” which could have included other school-and community- 
sponsored afterschool activities. The study was conducted in an elementary school in an urban district. It appears that the study 
was conducted in 2011-12, although the implementation year is not reported directly. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome 5 ‘ or, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
SSAT - reading Tier Il Tier Ill Grades 3-4 40 0.95 (0.33)* 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III | 0s 


* Statistically significant at p<.05 


CITATION: Gardner, L. T. (2014). The impact of afterschool tutoring on reading scores of elementary students (Unpublished doctoral dissertation). 
Walden University, Minneapolis, MN. 
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ACADEMIC PROGRAMS 


Middle School Academic Intervention Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Middle School Academic Intervention Program targeted Location 
low-achieving middle grades students in two middle schools in LOCALE REGION 
urban California during 2009-10. Certified teachers designed 
lessons in response to areas of academic need for classes 
organized by student performance levels. The program was 
offered for the full school year, five days per week, for 1.5 hours H | 
each day. 
Urban West 

EFFECTIVENESS AT A GLANCE 

Participants 
Positive Effect CLASS GRADE TARGET STUDENTS 
The study found at least one improved outcome with no 

Grades 6-8 Low-Achieving Students 


overriding contrary evidence for 
Reading/ELA Achievement 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Negative Effect 


The study found at least one negative outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


: : om 3 Not Reported 67% Latino 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, Attendance & Enrollment, School Engagement, Science 25% Black 
Achievement, and Social & Emotional Competencies 8% Asian 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: Full school year, 5 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | MIDDLE SCHOOL ACADEMIC INTERVENTION PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool program for students with low test scores in two urban middle schools in 
California. Outcomes for students who participated were compared to outcomes for a matched group of non-participants. The compari- 
son students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 
The study was conducted in 2009-10. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ‘ ‘ a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Renaissance Star 
co co Grades 6-8 178 -0.35 (0.15)* 
Math score 
Renaissance Star 
co co Grades 6-8 178 -0.42 (0.15)* 


Math gain 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


* Statistically significant at p<.05 
co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome ¥ ny Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 

Renaissance Star . : 

: Tier Il Tier Ill Grades 6-8 178 0.43 (0.15)* 
Reading score 

Renaissance Star : : 

Tier Il Tier Ill Grades 6-8 178 0.65 (0.15)* 


Reading gain 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


CITATION: Jones, B. R. (2014). An afterschool program and its effect on the math and reading performance levels of the standardized testing and 
reporting (STAR) for identified at-risk students (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3609435) 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Los Angeles - Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Los Angeles aimed Location 


to increase student achievement in math and/or reading for LOCALE REGION 
students in low-performing Title | schools. Programs provided 

tutoring in math and/or reading. Services were targeted to any 

students not scoring proficient on standardized tests across 

the district. Tutoring during 2009-10 took place online or in H | 

small groups and was offered by state-approved providers, 


including non-profit, for-profit, or faith-based organizations. 


Urban West 

= od od =O AYA =I | 0-3 Wc aN ed 

Participants 

CLASS GRADE TARGET STUDENTS 
Positive Effect 

- Low-Achieving Student 
The study found at least one improved outcome with no GENES A: sii uae ai 
overriding contrary evidence for 
Reading/ELA Achievement SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 100% Low-Income 87% Latino 3% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & 

Graduation, Attendance & Enrollment, School Engagement, Science 8% Black 2% White 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Not reported 
DURATION: Full school year, average attendance=26 hours 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (LOS ANGELES - MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on mathematics and English language arts achievement of supplemental educational 
services for middle grades students in Los Angeles Unified School District. Outcomes for students who participated were compared 
to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which could have 
included other school- and community-sponsored afterschool activities. The study was conducted in 2009-10. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome i r ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards Middl 
§ § age 9,581 0.01 (0.02) 


Test - mathematics grades 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome : ; or Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards Middl 
: 7 . I e 
Test - English 5 .02)* 
g Tier Ill Tier Ill grades 9,574 0.06 (0.02) 


language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


CITATION: Barnhart, M. K. (2011). The impact of participation in supplemental educational services (SES) on student achievement: 2009-10 (Publication 
No. 379). Los Angeles, CA: Los Angeles Unified School District Research Unit. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 43 


ACADEMIC PROGRAMS 


Warrior After School 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Warrior After School program, offered in one middle Location 
school in Georgia from 2004-05 to 2007-08, aimed to improve LOCALE REGION 
the academic performance of academically at-risk middle 
school students. Certified teachers provided afterschool 
instruction to small groups of students. The study does not 
provide more detail about the content of the program. During 
2004-05, 2005-06, and 2007-08, the program was offered 
from October through April, two days per week, for 1.5 hours Not Reported Sactheast 
each day. In 2006-07, the program ran only from January 
through April. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Middle Grades Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


9 a 0, A 9 j 
ment, Other Achievement, Physical Activity/Health, Promotion & 53% Low-Income 53% White 2% Asian 
Graduation, Attendance & Enrollment, School Engagement, Science 36% Black 3% Other 
Achievement, and Social & Emotional Competencies 6% Latino 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: 7 months, 2 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 44 


ACADEMIC PROGRAMS | WARRIOR AFTER SCHOOL 


Study Details 


STUDY DESCRIPTION 


This quasi-experiment estimates the impact of Warriors After School for middle school students in one school in Georgia. Outcomes for 


students who participated were compared to outcomes for a matched group of non-participants. The comparison students experienced 


“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study was conducted 


in 2004-08. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . i WEG: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced : : Middle 
Tier Il Tier Il 510 0.49 (0.09)* 
Competency Test - school ( ) 
mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 
* Statistically significant at p<.05 
READING/ELA ACHIEVEMENT 
ESSA evidence tier 
Number of Effect size 
Outcome ¥ a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced : : Middle 
Tier Il Tier Il 510 0.99 (0.09)* 
Competency Test - school ( ) 
reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


CITATION: Little, K. N. (2009). Effects of an intervention afterschool program on academic achievement among middle school students (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3342460) 
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MULTI-STUDY PROGRAM SUMMARY 


Supplemental Educational Services (Chicago) 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


Supplemental Educational Services in Chicago Positive Effect 
aimed to increase student achievement in math Overall, the studies found at least one improved 
and/or reading for students in low-performing outcome with no overriding contrary evidence for 


Title | schools. The services assessed in this study 


were offered to students in grades 3-8 who were Mathie Wialice Achievement 


not proficient on state standardized tests. The Reading/ELA Achievement 
program provided tutoring and other educational 
interventions in math and/or reading offered by 


state-approved providers. 


This review identified three studies of this program. 
The findings from each of these studies are presented individually on pages 47 — 53 of this document. 


¢ Heinrich, C. J., Burch, P., Good, A., Acosta, R., Cheng, H., Dillender, M.,. . ., Stewart, M. (2014). Improving the 
implementation and effectiveness of out-of-school-time tutoring. Journal of Policy Analysis and Management, 
33(2), 471-494. 


« Jones, C. J. (2009). The 2008 supplemental educational services program: Year 5 summative evaluation 
(Unpublished report). Chicago, IL: Chicago Public Schools Office of Extended Learning Opportunities. 


« Office of Research, Evaluation, and Accountability. (2007). SES tutoring programs: An evaluation of year 3 
in the Chicago Public Schools (Unpublished report). Chicago, IL: Chicago Public Schools Office of Extended 


Learning Opportunities. 


MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (CHICAGO) | SUMMARY #1 


Supplemental Educational Services (Chicago — 2005-06) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Chicago aimed to Location 


increase student achievement in math and/or reading for LOCALE REGION 
students in low-performing Title | schools. The services 

assessed in this study were offered during 2005-06 to 

students in grades 3-8 who were not proficient on state 

standardized tests. The program provided tutoring and H | 

other educational interventions in math and/or reading 


offered by state-approved providers. 


Urban Midwest 

EFFECTIVENESS AT A GLANCE 

Participants 

CLASS GRADE TARGET STUDENTS 
Positive Effect 
The study found at least one improved outcome with no Grades 3-8 Low-Achieving Students 
overriding contrary evidence for 

Reading/ELA Achievement SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
100% Low-I V V i 
ment, Other Achievement, Physical Activity/Health, Promotion & Mae 56% Black 2% White 


Graduation, Attendance & Enrollment, School Engagement, Science 42% Latino 1% Asian 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff 
DURATION: Full school year, 30-80 hours offered across sites 


IMPLEMENTATION SUPPORT: None reported 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (CHICAGO) | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impacts of Supplementary Educational Services for Chicago Public Schools students in grades 
3-8 who scored at or below the 50th percentile on the state reading or math assessment and who received at least 30 hours of the 
intervention. The comparison group consists of a matched group of students who were eligible for but did not participate in the 
program. The comparison students experienced “business as usual,” which could have included other school-and community- 
sponsored afterschool activities. The study was conducted in Chicago Public Schools during 2005-06. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome x 5 eo Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Illinois Standards 


: Cannot 
GHG gn: - 8 8 Grades 3-8 66,088 determine 
mathematics 


: 3 , : Cannot 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; Pa Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Illinois Standards 
Achievement Test - Tier III Tier Ill Grades 3-11 68,757 eanaet 
, determine 


reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III catinet 


determine 


* Statistically significant at p<.05 


CITATION: Office of Research, Evaluation, and Accountability. (2007). SES tutoring programs: An evaluation of year 3 in the Chicago Public Schools 
(Unpublished report). Chicago, IL: Chicago Public Schools Office of Extended Learning Opportunities. 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (CHICAGO) | SUMMARY #2 


Supplemental Educational Services (Chicago - 2007-08) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Chicago aimed to Location 


increase student achievement in math and/or reading for LOCALE REGION 
students in low-performing Title | schools. The services 

assessed in this study were offered during 2007-08 to 

students in grades 3-8 who were not proficient on state 

standardized tests. The program provided tutoring and i | 

other educational interventions in math and/or reading 


offered by state-approved providers. 


Urban Midwest 

EFFECTIVENESS AT A GLANCE 

Participants 

CLASS GRADE TARGET STUDENTS 
Positive Effect 
The study found at least one improved outcome with no Grades 3-8 Tow-Achieving Students 
overriding contrary evidence for 

Mathematics Achievement SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, = 9 9 ‘ 
ment, Other Achievement, Physical Activity/Health, Promotion & 100% Low-Income 61% Black 1% White 
Graduation, Attendance & Enrollment, School Engagement, Science 37% Latino 1% Other 
Achievement, and Social & Emotional Competencies 1% Asian 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Program staff 
DURATION: Full school year, 30-80 hours offered across sites 


IMPLEMENTATION SUPPORT: None reported 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (CHICAGO) | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of Supplementary Educational Services for Chicago Public Schools students in grades 
3-8 who received these services in 2007-08. Outcomes for students who participated were compared to outcomes for a matched 
group of non-participants. The comparison students experienced “business as usual,” which could have included other school- and 


community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 


Outcome ; ; eae Group 
Without site and With site and students (Standard error) 


sample criteria sample criteria 


Illinois Standards 
Achievement Test - Tier Ill Tier Ill Grades 3-8 57,923 0.04 (0.01)* 
mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . fl Pe: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Illinois Standards 
Achievement Test - 8 8 Grades 3-8 58,079 0.02 (0.01) 


reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Jones, C. J. (2009). The 2008 supplemental educational services program: Year 5 summative evaluation (Unpublished report). Chicago, IL: 
Chicago Public Schools Office of Extended Learning Opportunities. 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (CHICAGO) | SUMMARY #3 


Supplemental Educational Services (Chicago - 2008-11) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Chicago aimed to Location 


increase student achievement in math and/or reading for stu- LOCALE REGION 
dents in low-performing Title | schools. The services assessed 

in this study were offered during 2008-11 to students in grades 

3-8 who were not proficient on state standardized tests. The 

program provided tutoring and other education interventions i | 

in math and/or language arts. The services were offered by the 


district and state-approved private providers. 


Urban Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 3-8 Low-Achieving Students 
Positive Effect SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEAS EEED FOR THIS INZERNENIAON: General mereNes 100% Low-Income 49% Latino 2% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & 

Graduation, Attendance & Enrollment, School Engagement, Science 47% Black 2% White 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help SETTING: School and community sites 
BEST PRACTICES USED: Positive adult- DELIVERY FORMAT: Group and one-to-one formats 


lationshi 
Stee ner clavensiipe STAFFING: Certified teachers, program staff 


DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (CHICAGO) | SUMMARY #3 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of Supplementary Educational Services for Chicago Public Schools students in grades 3-8 
who received these at any time during 2008-11. Outcomes for students who participated were compared to outcomes for a matched 
group of non-participants. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome r 4 ae Group 
Without site and With site and students (Standard error) 


sample criteria sample criteria 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
mathematics 


Tier Ill Tier Ill Year 1 61,464 0.05 (0.01)* 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
mathematics 


Tier Ill Tier III Year 2 63,773 0.05 (0.01)* 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
mathematics 


Tier Ill Tier Ill Year 3 204,094 0.06 (0.00)* 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
mathematics 


Tier Ill Tier Ill Year 4 68,411 0.05 (0.01)* 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill | 00s 


* Statistically significant at p<.05 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (CHICAGO) | SUMMARY #3 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome Group 


Without site and With site and students (Standard error) 
sample criteria sample criteria 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
reading 


Tier Ill Tier Ill Year 1 61,171 0.04 (0.01)* 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
reading 


Tier Ill Tier Ill Year 2 63,506 0.09 (0.01)* 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
reading 


Tier Ill Tier Ill Year 3 205,187 0.08 (0.00)* 


Illinois Standards 
Achievement Test/lowa 
Test of Basic Skills - 
reading 


Tier III Tier Ill Year 4 68,541 0.04 (0.01)* 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


CITATION: Heinrich, C. J., Burch, P., Good, A., Acosta, R., Cheng, H., Dillender, M., .. ., Stewart, M. (2014). Improving the implementation and 
effectiveness of out-of-school-time tutoring. Journal of Policy Analysis and Management, 33(2), 471-494. 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Dallas) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services, offered in Dallas, Texas Location 


from 2008-11, aimed to increase student achievement in math LOCALE REGION 
and/or reading for students in low-performing Title | schools. 

The services assessed in this study were offered to students 

in grades 3-8 who were not proficient on state standardized 

tests. The program provided tutoring and other education i | 

interventions in math and/or language arts. The services were 


offered by the district and state-approved private providers. 


Urban Southwest 

The program was offered during the full school year, and the 
average participant received 22 hours 
of tutoring. 

Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE 

Grades 3-8 Low-Achieving Students 

Mixed Effects 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 
The study found a mix of improved and null or negative 


outcomes for outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 

0, = 9 ‘ 9 q 
ment, Other Achievement, Physical Activity/Health, Promotion & 70% Low-Income 63% Latino 4% White 
Graduation, Attendance & Enrollment, School Engagement, Science 32% Black 1% Asian 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help SETTING: School and community sites 
BEST PRACTICES USED: Positive adult- DELIVERY FORMAT: Group and one-to-one formats 


student relationships 
2 STAFFING: Certified teachers, program staff 


DURATION: Full school year, 1 hour/day, average 
attendance=22 sessions 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (DALLAS) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool tutoring programs on mathematics and reading outcomes for students in 
Dallas, Texas. The comparison group is comprised of students eligible for afterschool tutoring who did not attend any tutoring sessions. 
The comparison students experienced “business as usual,” which could have included other school- and community-sponsored after- 
school activities. The study examined impacts for four years (2008-11). 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome Group 


Without site and With site and students (Standard error) 
sample criteria sample criteria 


Texas Assessment 
of Knowledge and 
Skills/State of Texas 
Assessments of 
Academic Readiness - 
mathematics 


§ § Year 1 9,294 -0.08 (.02)* 


Texas Assessment 
of Knowledge and 
Skills/State of Texas 
Assessments of 
Academic Readiness - 
mathematics 


Tier Ill Tier Ill Year 2 13,806 0.13 (.02)* 


Texas Assessment 
of Knowledge and 
Skills/State of Texas 
Assessments of 
Academic Readiness - 
mathematics 


8 8 Year 3 13,332 0.02 (.02) 


Texas Assessment 
of Knowledge and 
Skills/State of Texas 
Assessments of 
Academic Readiness - 
mathematics 


Tier Ill Tier Ill Year 4 14,360 0.05 (0.02)* 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill | 003 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 55 


ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (DALLAS) 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome A . os Group 
Without site and With site and students (Standard error) 


sample criteria sample criteria 


Texas Assessment 
of Knowledge 
and Skills/State of 
Texas Assessments 
of Academic 
Readiness - reading 


§ § Year 1 9,294 -0.11 (0.02)* 


Texas Assessment 
of Knowledge and 
Skills/State of 
Texas Assessments 
of Academic 
Readiness - reading 


Tier Ill Tier Ill Year 2 14,06 0.11 (0.02)* 


Texas Assessment 
of Knowledge 
and Skills/State of 
Texas Assessments 
of Academic 
Readiness - reading 


§ § Year 3 13,428 0.02 (0.02) 


Texas Assessment of 

Knowledge and Skills/ 

State of Texas Assess- 8 8 Year 4 14,670 0.01 (0.02) 
ments of Academic 
Readiness - reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Heinrich, C. J., Burch, P., Good, A., Acosta, R., Cheng, H., Dillender, M., . . ., Stewart, M. (2014). Improving the implementation and 
effectiveness of out-of-school-time tutoring. Journal of Policy Analysis and Management, 33(2), 471-494. 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Florida) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Florida aimed to Location 


increase academic achievement in math and/or reading for LOCALE REGION 
students in grades 4-10 in low-performing Title | schools. This 

study reports program impacts for six large Florida districts 

during 2006-07. Programs provided tutoring in math and/or 

reading, but the study reports few details about the content or i | 

format of the tutoring. Tutoring was offered for the full school 


year, for up to one hour per day. 


Urban Southeast 

= fol Oe AYA ee We cd ea 

Participants 

CLASS GRADE TARGET STUDENTS 
Positive Effect 
The study found at least one improved outcome with no Grades 4-10 Tow-Achieving Students 
overriding contrary evidence for 

Mathematics Achievement SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 
The study found neither improved nor negative outcomes for 
Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, = 
ment, Other Achievement, Physical Activity/Health, Promotion & 30% Low-Income Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers, volunteers 
DURATION: Full school year, 30-60 minutes/day 


IMPLEMENTATION SUPPORT: Provider training 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (FLORIDA) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on mathematics and reading achievement of supplemental educational services for 
students in grades 4-10 in six large Florida school districts. Outcomes for students who participated were compared to outcomes 
for a matched group of non-participants. The comparison students experienced “business as usual,” which could have included 
other school- and community-sponsored afterschool activities. The study was conducted in 2006-07. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . " Senate Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Florida Comprehensive 
Assessment Test - Tier Ill Tier Ill Grades 4-10 3,130 0.12 (0.03)* 
mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; aT Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Florida Comprehensive 
Assessment Test - 8 8 Grades 4-10 12,720 0.03 (0.02) 


reading 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Allen, M. L. (2008). Assessing the effectiveness of supplemental education services in urban Florida school districts (Doctoral dissertation). 
Retrieved from http://ufdcimages.uflib.ufl.edu/UF/E0/02/19/69/00001/allen_m.pdf 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Minneapolis) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Minneapolis, Minnesota, Location 

offered in 2008-11, aimed to increase student achievement LOCALE REGION 

in math and/or reading for students in low-performing Title 

| schools. The services assessed in this study were offered 

to students in grades 3-8 who were not proficient on state 

standardized tests. The program provided tutoring and other i | 

education interventions in math and/or language arts. The 

services were offered by the district and state-approved Watraxe Midwest 
private providers. The program was offered for the full school 

year, and the average participant received 28 hours of tutoring. 


Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 3-8 Low-Achieving Students 


Mixed Effects 


The study found a mix of improved and null or negative 
SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 9906 Low-laconié 47% Black 9% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & 

Graduation, Attendance & Enrollment, School Engagement, Science 28% Latino 7% Other 
Achievement, and Social & Emotional Competencies 9% White 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Tutoring, homework help SETTING: School and community sites 

BEST PRACTICES USED: Positive adult- DELIVERY FORMAT: Group and one-to-one formats 


ent relationships 
Studie nbrelavenslip STAFFING: Certified teachers, program staff 


DURATION: Full school year, 1 hour/day, average 
attendance=28 sessions 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (MINNEAPOLIS) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool tutoring programs on mathematics and reading outcomes for students in 
Minneapolis, Minnesota. The comparison group is comprised of students eligible for afterschool tutoring who did not attend any 
tutoring sessions. The study examines impacts for four years (2008-11). 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome - ; rae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Minnesota 
Comprehensive 
Assessments II — 
mathematics 


8 8 Year 1 1,400 -0.01 (.05) 


Minnesota 
Comprehensive 
Assessments II — 
mathematics 


8 8 Year 2 788 -0.01 (.07) 


Minnesota 
Comprehensive 
Assessments II — 
mathematics 


Tier Ill Tier Ill Year 3 5,044 0.19 (.03)* 


Minnesota 
Comprehensive 
Assessments II — 
mathematics 


8 8 Year 4 4,298 0.05 (.03) 


Average effect size for Mathematics Achievement outcomes at Tiers I-III | 00s 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (MINNEAPOLIS) 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome A . oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Minnesota 
Comprehensive 
Assessments II — 
reading 


§ § Year 1 2,862 -0.20 (0.04)* 


Minnesota 
Comprehensive 
Assessments II — 
reading 


§ § Year 2 1,602 -0.20 (0.05)* 


Minnesota 
Comprehensive 
Assessments II — 
reading 


Tier Ill Tier Ill Year 3 5,024 0.14 (0.03)* 


Minnesota 
Comprehensive 
Assessments II — 
reading 


8 8 Year 4 4,246 -0.04 (0.03) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III -0.07 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Heinrich, C. J., Burch, P., Good, A., Acosta, R., Cheng, H., Dillender, M., .. ., Stewart, M. (2014). Improving the implementation and effective- 
ness of out-of-school-time tutoring. Journal of Policy Analysis and Management, 33(2), 471-494. 
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ACADEMIC PROGRAMS 


Academic Volunteer Mentor Service Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Academic Volunteer Mentor Service program, offered Location 


in three high schools in California in 1999, was a one-on-one LOCALE REGION 
mentoring program that matched volunteer mentors with 

at-risk students to improve academic performance and prevent 

school dropout. Mentors from the local community provided 

support and guidance, particularly focused on academic and i | 

behavioral issues. Mentoring was short-term (4-18 months), and 

students were expected to participate for at least 16 hours in 


Urban West 
three consecutive months. 


EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 


Students at Risk of 


Positive Effect Grades 9-12 Academic Failure & Dropout 


The study found at least one improved outcome with no 
overriding contrary evidence for 

Attendance & Enrollment SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
General Achievement 


Reading/ELA Achievement 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Not Reported 35% White 6% Latino 

18% Black 30% Other 
8% Asian 

NOT MEASURED FOR THIS INTERVENTION: General Achievement, 

Physical Activity/Health, Promotion & Graduation, School Engagement, 

Science Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: Mentoring SETTING: Not reported 

BEST PRACTICES USED: Positive adult- DELIVERY FORMAT: Individualized, one-to-one format 


Shigeptrelaenships STAFFING: Volunteers 


DURATION: Full school year, 2 sessions/month for up to 6 hours 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | ACADEMIC VOLUNTEER MENTOR SERVICE PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experimental study estimates the impact of the Academic Volunteer Mentor Service Program for students in three high 
schools in Alameda, California. Outcomes for students who participated were compared to outcomes for a matched group of non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. The study was conducted in three high schools in 2009-10. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome , e a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school : : 
Tier Ill Tier Ill Grades 9-12 122 3.73 (0.30)* 
days absent 
Average effect size for Attendance & Enrollment outcomes at Tiers I-IIl 3.73 
* Statistically significant at p<.05 
GENERAL ACHIEVEMENT 
ESSA evidence tier 
Number of Effect size 
Outcome ; ; ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Grade point average Tier Ill Tier Ill Grades 9-12 122 0.65 (0.19)* 


Average effect size for General Achievement outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome " i ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Stanford Achievement 
8 8 Grades 9-12 122 0.33 (0.18) 


Test 9 — mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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ACADEMIC PROGRAMS | ACADEMIC VOLUNTEER MENTOR SERVICE PROGRAM 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome 5 5 ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Stanford Achievement ? 7 
Tier Ill Tier Ill Grades 9-12 122 0.84 (0.19)* 
Test 9 - language 
Stanford Achievement : : 
: Tier Ill Tier Ill Grades 9-12 122 0.71 (0.19)* 
Test 9 - reading 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.78 


* Statistically significant at p<.05 


CITATION: Starks, F. I., Jr. (2002). Mentoring at-risk youth: An intervention for academic achievement (Doctoral dissertation). Retrieved from ProQuest 
Dissertations and Theses Database (UMI No. 3053015) 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Los Angeles - High School) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Los Angeles aimed Location 


to increase student achievement in math and/or reading for LOCALE REGION 
students in low-performing Title | schools. Programs provided 

tutoring in math and/or reading for students in grades 9-12. 

Services were targeted to any students not scoring proficient 

on standardized tests across the district. In 2009-10, tutoring i | 

took place online or in small groups and was offered by 

state-approved providers including non-profit, for-profit, or 


Urban West 
faith-based organizations. 


EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 


Positive Eflect Grades 9-12 Low-Achieving Students 


The study found at least one improved outcome with no 


overriding contrary evidence for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Reading/ELA Achievement 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 100% Low-Income 87% Latino 3% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, Attendance & Enrollment, School Engagement, Science 8% Black 2% White 


Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Not reported 
DURATION: Full school year, average attendance=26 hours 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (LOS ANGELES - HIGH SCHOOL) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on mathematics and English language arts achievement of supplemental educational 
services for high school students in Los Angeles Unified School District. Outcomes for students who participated were compared 
to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which could have 
included other school- and community-sponsored afterschool activities. The study was conducted in 2009-10. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome . . ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards F 
8 8 High school 3,611 0.05 (0.03) 


Test - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome ; 5 nar Group 
Without site and With site and students (Standard error) 


sample criteria sample criteria 


California Standards 
Test - English Tier Ill Tier Ill High school 2,585 0.08 (0.04)* 
language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.08 


* Statistically significant at p<.05 


CITATION: Barnhart, M. K. (2011). The impact of participation in supplemental educational services (SES) on student achievement: 2009-10 (Publication 
No. 379). Los Angeles, CA: Los Angeles Unified School District Research Unit. 
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ACADEMIC PROGRAMS 


21st Century Community Learning Centers (Georgia — Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This 21st Century Community Learning Center program, Location 
offered in two rural schools in North Georgia during 2010-11, LOCALE REGION 
aimed to improve math achievement of low-income youth 
in grades 3-5. The program was staffed by certified teachers, 
but the study provides no details about the math intervention 
or its duration. 
Rural Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 3-5 Low-Achieving Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 0 0 . 
Graduation, Reading/ELA Achievement, Attendance & Enrollment, 81% Low-Income 93% White 
School Engagement, Science Achievement, and Social & Emotional 5% Latino 
Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers, program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (GEORGIA — ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool program for students identified as at risk of mathematics failure in 
two rural schools in Georgia. Outcomes for students who participated were compared to outcomes for a matched group of non- 
participants. The comparison students experienced “business as usual,” which could have included other school-and community- 
sponsored afterschool activities. The study was conducted in 2010-11. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ’ Nee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced 
8 8 Grades 3-5 14 0.24 (0.19) 


Competency Test - 
mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Hobbs, C. L. (2012). Effects of an afterschool program on elementary and middle school math achievement in Georgia schools (Unpublished 
doctoral dissertation). Liberty University, Lynchburg, VA. 
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ACADEMIC PROGRAMS 


Afterschool Literacy Program with Barton Reading & Spelling System® 


ea COXCH VP) 104-0 ed STUDY LOCATION AND PARTICIPANTS 


This afterschool literacy program, offered in one school district Location 
in Arkansas, aimed to improve the reading and spelling skills LOCALE REGION 
of students in grades 3-6 with reading skills below grade 
level. The program offered tutoring using the Barton Reading 
& Spelling System®, a multisensory intervention through 
which students move at their own pace. Tutoring was offered 
for the full school year, two days per week, and was delivered 
by trained school personnel as well as students from a local 


Rural Southeast 
university. It appears that the program was offered during 2013-14. 


Participants 


CLASS GRADE TARGET STUDENTS 


ee Students Reading 
EFFECTIVENESS AT A GLANCE TAGES Below Grade Level 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: Attendance & 
Enrollment, General Achievement, Other Achievement, Physical 
Activity/Health, Promotion & Graduation, Reading/ELA Achievement, 
School Engagement, Science Achievement, and Social & Emotional 
Competencies 


Not Reported 100% White 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENT: Tutoring using the Barton Reading SETTING: School site 


] ® 
and spelling: ysiem DELIVERY FORMAT: Individualized, one-to-one format 


BEST PRACTICES USED: Sequenced 
oe STAFFING: Certified teachers, specialized staffing 
DURATION: Full school year, 2 days/week 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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ACADEMIC PROGRAMS | AFTERSCHOOL LITERACY PROGRAM WITH BARTON READING & SPELLING SYSTEM® 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on mathematics achievement of an afterschool program using the Barton Reading & 
Spelling System® for students in grades 3-6 in one school district in Arkansas. Outcomes for students who participated were compared 
to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which could have 
included other school- and community-sponsored afterschool activities. It appears that the study was conducted in 2013-14. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Arkansas Comprehensive 
Testing, Assessment, 


and Ability Program foe) oo Grades 3-6 38 0.24 (0.33) 
Benchmark — mathematics 


—1year follow-up 


Arkansas Comprehensive 
Testing, Assessment, 


and Ability Program oo co Grades 3-6 38 0.37 (0.33) 


Benchmark — mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill | os 


—1year follow-up 
co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Watkins, T. R. (2016). The influence of literacy intervention on mathematics (Doctoral dissertation). Retrieved from ProQuest Dissertations 
and Theses Database (UMI No. 10241100) 
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ACADEMIC PROGRAMS 


Leap Track 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The LeapTrack Assessment and Instruction System was Location 
designed to improve reading skills of students in grades LOCALE REGION 
3-5 through a commercial, computer-based supplemental 
reading program. The program, offered in one school in 
Tennessee in 2006-07, asked students to listen individually 
to a story recitation, then read the story on their own, read 
the story to their teacher, and respond to higher-order 
questions about the text. The program was offered for eight 


Rural Southeast 
months, two to three days per week. 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 3-5 Low-Achieving 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement 
: : : 9 - 87% Whit 
Physical Activity/Health, Promotion & Graduation, Attendance & 60% Low-Income ° © 


Enrollment, School Engagement, Science Achievement, and 12% Black 
Social & Emotional Competencies 1% Latino 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Individualized, one-to-one format 


STAFFING: Certified teachers 
DURATION: 8 months, 2-3 days/week 


IMPLEMENTATION SUPPORT: Program manual 
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ACADEMIC PROGRAMS | LEAPTRACK 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact on reading achievement of LeapTrack for students in grades 3-5 who attended 30 or more 


days of the program. Outcomes for students who participated were compared to outcomes for a matched group of non-participants. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 
activities. The study was conducted in one school in Tennessee during 2006-07. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 

Outcome ; ' we Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 

Tennessee 
Comprehensive 
co co Grades 3-5 60 0.01 (0.26) 
Assessment Program - 
reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: Armstrong, B. L. P. (2007). Evaluating the impact of the LeapTrack Assessment and Instruction System through an afterschool reading 
intervention class (Doctoral dissertation, The University of Mississippi). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3282864) 
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ACADEMIC PROGRAMS 


Math Academic Enhancement Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Math Academic Enhancement Program, offered in five Location 
schools in New York state in 2001-02, was designed to improve LOCALE REGION 
the math performance of fourth grade students who were 
below proficiency on state standardized math assessments. 
The program, taught by certified classroom teachers, directed 
teachers to ask higher-order, critical thinking questions to 
bolster students’ abilities to demonstrate verbally and in 
writing their comprehension of mathematical processes. The Le aoe es Nordieact 
program was offered for 12 weeks, two days per week, for 
approximately one hour each day. 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grade 4 Low-Achieving Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Other Achievement, Physical Activity/Health, 


0 - 
Promotion & Graduation, Reading/ELA Achievement, Attendance & 47% Low-Income Not Reported 
Enrollment, School Engagement, Science Achievement, and Social 
& Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: 12 weeks, 2 days/week, 1.25 hours/day 


IMPLEMENTATION SUPPORT: Provider training 
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ACADEMIC PROGRAMS | MATH ACADEMIC ENHANCEMENT PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the Math Academic Enhancement Program on math achievement for fourth grade 
students who did not achieve mastery level on a third-grade state standardized assessment. Outcomes for students who participated 
were compared to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which 
could have included other school- and community-sponsored afterschool activities. The study was conducted in five schools in one 
district in Nassau County, New York, in 2001-02. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; Vike, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Test of New York 
State Standards - 8 8 Grade 4 488 0.06 (0.09) 


mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Deeb-Westervelt, W. (2003). The effects of an afterschool academic intervention services math program on the grade four New 
York state mathematics assessment: A quasi-experimental case study (Doctoral dissertation). Retrieved from ProQuest Dissertations and 
Theses Database (UMI No. 3088552) 
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ACADEMIC PROGRAMS 


Math Program with ST Math 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This program, offered in one school in Orange County, Location 
California during 2011-12, targeted students in grades 3-5 who LOCALE REGION 
were below proficiency in math on state standardized tests and 

used a spatial temporal (ST) approach to teaching mathemat- 

ics. ST Math is based on mastery learning theory and includes 

the full math curriculum and interactive software. Students also 

received homework assistance. The program was offered for 


12 weeks, one day per week, for 45 minutes. Sita West 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 3-5 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, Reading/ELA Achievement, Attendance & Enrollment, 
School Engagement, Science Achievement, and Social & Emotional 
Competencies 


Not Reported Not Reported 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 


BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 
STAFFING: Certified teachers, specialized staffing, program staff 
DURATION: 12 weeks, 1 day/week, 45 minutes/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | MATH PROGRAM WITH ST MATH 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on math achievement of a spatial temporal math software program for students in grades 
3-5 in one school in Orange County, California. Outcomes for participants were compared to outcomes for non-participants, some of 
whom participated in an afterschool homework group. The study was conducted in 2011-12. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i : ays Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
District benchmark 
mathematics 8 8 Grade 3 29 -0.76 (0.39) 
assessment 
District benchmark 
mathematics 8 8 Grade 4 16 -0.93 (0.53) 


assessment 
Average effect size for Mathematics Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Tom, D. M. (2012). Impact on achievement with ST math after school instruction (Doctoral dissertation). Retrieved from ProQuest Disserta- 
tions and Theses Database (UMI No. 3530680) 
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ACADEMIC PROGRAMS 


Purple Sage Elementary Writing Project 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Purple Sage Elementary Writing Project, offered in one Location 
school in Houston, Texas in 2003, focused on the development LOCALE REGION 
of creativity and student voice through writing with a popu- 

lation of predominantly low-income fourth grade students. 

The program engaged students in multiple genres of creative 

writing and integrated visual and performing arts. Activities i | 

included the creation of comic books, brochures, and poetry 


and writing and performing skits. The program was offered for 


Urban Southwest 
five months, one day per week, for 1.5 hours per day. 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grade 4 Low-Income 
SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


0, 
Physical Activity/Health, Promotion & Graduation, Attendance & Not Reported 48% Black 
Enrollment, School Engagement, Science Achievement, and 43% Latino 
Social & Emotional Competencies 10% Asian 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff 
DURATION: 5 months, 1 day/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual 
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ACADEMIC PROGRAMS | PURPLE SAGE ELEMENTARY WRITING PROJECT 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of attending an afterschool writing program on a state standardized writing assessment. 
Outcomes for the 21 students who participated were compared to outcomes for a matched group of non-participants. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 
The study was conducted in Houston, Texas in 2003. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i z ea Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Texas Assessment EI t 
ementary 
of Knowledge and 42 .24 (0.31 
g 8 8 students Oo2a 24) 


Skills - writing 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Phillips, T. D. (2004). The effectiveness of writing voice in an afterschool writing program for fourth-graders (Master’s thesis). Retrieved from 
ProQuest Dissertations and Theses Database (UMI No. 1423877) 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Los Angeles - Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services in Los Angeles aimed to Location 
increase achievement in math and/or reading for students in LOCALE REGION 
low-performing Title | schools. Programs provided tutoring in 

math and/or reading. Services during 2009-10 

were targeted to any students not scoring proficient on 

standardized tests across the District. Tutoring took place i | 

online or in small groups and was offered by state-approved 


providers, including non-profit, for-profit, or faith-based 


Urban West 

organizations. The program was offered for the full school year 
and, on average, participants attended 26 hours of tutoring. 

Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 1-5 Low-Achieving Students 

SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 


0, a 9 . 9 ; 
Graduation, Attendance & Enrollment, School Engagement, Science 100% Low-Income 87% Latino 3% Asian 
Achievement, and Social & Emotional Competencies 8% Black 2% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Individualized, one-to-one format 


STAFFING: Not reported 
DURATION: Full school year, average attendance=26 hours 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (LOS ANGELES - ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on mathematics and English language arts achievement of supplemental educational 
services for elementary school students in Los Angeles Unified School District. Outcomes for students who participated were compared 
to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which could have 
included other school- and community-sponsored afterschool activities. The study was conducted in 2009-10. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome . . ay Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards Elemen 
§ § emeltaly 6,076 0.03 (0.03) 


Test - mathematics students 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome ; : an Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards EI i 
: ementary 
Test - English E ; 
g § § ae 9,922 0.03 (0.02) 


language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Barnhart, M. K. (2011). The impact of participation in supplemental educational services (SES) on student achievement: 2009-10 (Publication 
No. 379). Los Angeles, CA: Los Angeles Unified School District Research Unit. 
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ACADEMIC PROGRAMS 


William Penn Initiative For Neighborhood Success 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The William Penn Initiative for Neighborhood Success program, Location 
a 21st Century Community Learning Center offered from 2002- LOCALE REGION 
04 in one district in southeast Pennsylvania, was designed to 

improve the math and reading achievement of low-income 

urban elementary school students. The program used supple- 

mental reading and math curricula and a basic skills curriculum i | 

called Foundations Achievement Math and English. Homework 


help was provided, and fifth grade students received tutoring 


Urban Northeast 

to prepare them for the state standardized tests. The program 
also included weekly enrichment clubs. The program was 
offered for the full school year, five days per week, three hours 
each day. Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades K-6 Low-Income 

SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Attendance & Enrollment 
General Achievement 
Mathematics Achievement 


Reading/ELA Achievement 


School Engagement 


Social & Emoti Ic fen 71% Low-Income 91% Black 1% Asian 
m n m n 
ocia otional Competencies 4% White 2% Other 
2% Latino 


NOT MEASURED FOR THIS INTERVENTION: Physical Activity/ 
Health, Promotion & Graduation, and Science Achievement 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework support SETTING: School site 
BEST PRACTICES USED: Family engagement DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers, program staff 
DURATION: Full school year, 5 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training, ongoing support 
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ACADEMIC PROGRAMS | WILLIAM PENN INITIATIVE FOR NEIGHBORHOOD SUCCESS 


Study Details 


STUDY DESCRIPTION 
This randomized controlled trial estimates the impact of an afterschool program offered during 2002-04 for students in grades K-6. The 


study was conducted in one elementary school in the William Penn School District in southeast Pennsylvania. Outcomes for students 
who were randomly selected for an offer to participate in the afterschool program are compared to outcomes for students who did not 
receive the offer. The comparison students experienced “business as usual,” which could have included other school- and communi- 


ty-sponsored afterschool activities. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome 5 . Coe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school 
8 8 Grades K-6 98 -0.23 (0.21) 


days absent 
Average effect size for Attendance & Enrollment outcomes at Tiers I-IIl 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Report card grade: 
: : § § Grades K-6 102 -0.29 (0.21) 
homework 
Average effect size for General Achievement outcomes at Tiers I-Ill -0.29 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
our G Number of Effect size 
CuSUIS Without site and With site and Mey students (Standard error) 
sample criteria sample criteria 
Mathematics 
8 8 Grades K-6 95 -0.06 (0.21) 
course grade 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl -0.06 


§ Meets Tier IIl standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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ACADEMIC PROGRAMS | WILLIAM PENN INITIATIVE FOR NEIGHBORHOOD SUCCESS 


Study Details (Continued) 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome - r oy Group 
Without site & sample With site & students (Standard error) 
criteria sample criteria 
Reading course grade ce} 8 Grades K-6 95 -0.14 (0.21) 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III -0.14 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCHOOL ENGAGEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : Tate Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Report card grade: 
BO ORe ts § § Grades K-6 94 -0.06 (0.21) 
academic behavior 
School attachment 
8 8 Grades K-6 53 -0.13 (0.28) 
(self-report) 
Average effect size for School Engagement outcomes at Tiers I-Ill -0.10 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SOCIAL & EMOTIONAL COMPETENCIES 


ESSA evidence tier 


Number of Effect size 
Outcome ' : . ; Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 

Academic focus 
8 8 Grades K-6 98 0.33 (0.20) 

(teacher report) 

Report card grade: 
Sok aes ene § § Grades K-6 94 0.00 (0.21) 
non-academic behavior 
Average effect size for Social & Emotional Competencies outcomes at Tiers I-Ill 0.16 


§ Meets Tier IIl standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Zief, S. G. (2005). A mixed-methods study of the impacts and processes of an afterschool program for urban elementary youth (Doctoral 
dissertation, University of Pennsylvania). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3165777) 
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ACADEMIC PROGRAMS 


21st Century Community Learning Centers (Georgia - Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This 21st Century Community Learning Center program, Location 
offered in two rural schools in North Georgia during 2010-11, LOCALE REGION 
aimed to improve math achievement of low-income youth in 
grades 6-8. The program was staffed by certified teachers, 
but the study provides no details about the math intervention 
or its duration. 
Rural Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
=I od od = OM AYA =) | = 3--W  d  g Grades 6-8 Low-Achieving Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 0 0 . 
Graduation, Reading/ELA Achievement, Attendance & Enrollment, 81% Low-Income 93% White 
School Engagement, Science Achievement, and Social & Emotional 5% Latino 
Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers, program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (GEORGIA - MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact of an afterschool program for students identified as at risk of mathematics failure in a school 
in Georgia during 2010-11. Outcomes for students who participated were compared to outcomes for a matched group of non-partici- 


pants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome , ; men Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 
Georgia Criterion- 

Referenced 
oo oo Grades 6-8 66 0.16 (0.25) 

Competency Test - 


mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: Hobbs, C. L. (2012). Effects of an afterschool program on elementary and middle school math achievement in Georgia schools 
(Unpublished doctoral dissertation). Liberty University, Lynchburg, VA. 
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ACADEMIC PROGRAMS 


PAR eGo tiigaGonviiiniing Melee eon 


(Pittsburgh Charter Schools - Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Pittsburgh’s 21st Century Community Learning Centers Location 


aimed to improve academic and behavioral outcomes of LOCALE REGION 
low-income youth. This program, offered in two charter schools 

in Pittsburgh during 2008-09, provided homework help and 

specialized instruction in areas of academic weakness. The 

program was offered for the full school year, five days per i | 


week, for 2.5 hours each day. 


Urban Northeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 6-8 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 


9 = 9 

Graduation, Attendance & Enrollment, School Engagement, Science 84% Low-Income 74% Black 

Achievement, and Social & Emotional Competencies 13% White 
13% Other 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Tutoring, homework help SETTING: School site 

BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff 
DURATION: Full school year, 5 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS 
(PITTSBURGH CHARTER SCHOOLS - MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool tutoring and homework help program in two charter schools in Pittsburgh, 
Pennsylvania. Outcomes for students who participated were compared to outcomes for a matched group of non-participants. The 
treatment group for mathematics is defined as students who attended at least 50 percent of the days when the program focus was 
mathematics. The treatment group for reading is defined as students who attended at least 50 percent of the days when the program 
focus was reading. The comparison students experienced “business as usual,” which could have included other school- and communi- 
ty-sponsored afterschool activities. The study was conducted in 2008-09. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . x ane Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
ASight Pennsylvania 
Core Standards 
Benchmark 8 8 Grades 6-8 35 -0.18 (0.36) 


Assessments - 
mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier IIl standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : F aneee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
A4Sight Pennsylvania 
Core Standards 
8 8 Grades 6-8 Al -0.29 (0.32) 


Benchmark 
Assessments - reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Dreyer, K. J. (2010). An examination of academic outcomes for students who attend a school-based afterschool program (Doctoral disserta- 
tion). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3417442) 
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ACADEMIC PROGRAMS 


Academic Intervention Services (New York) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This program, offered in one middle school on Long Island, Location 
New York during 1999-2001, was designed for students LOCALE REGION 
identified by their teachers as at risk of not passing the eighth 

grade English Regents exam. Teachers offered supplementary 

instruction in English language arts using a curriculum they 

designed, with extensive use of the Day Book of Critical 

Reading and Writing. The program was offered for five weeks, 


two days per week, for one hour each day. Sita Nopihease 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grade 8 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical 
Activity/Health, Promotion & Graduation, Attendance & Enrollment, 
School Engagement, Science Achievement, and Social & Emotional 
Competencies 


2% Low-Income 95% White 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Supplemental instruction, tutoring SETTING: School site 


BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 
STAFFING: Certified teachers 
DURATION: 5 weeks, 2 days/week, 1 hour/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | ACADEMIC INTERVENTION SERVICES (NEW YORK) 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment compared outcomes for students who participated were compared to outcomes for a matched group of non-par- 
ticipants. The comparison students experienced “business as usual,” which could have included other school- and community-spon- 


sored afterschool activities. The study was conducted during the 1999-2000 and 2000-01 school years in one middle school located on 
Long Island in New York. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . y Weg: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
New York State 
English language 8 8 Grade 8 157 0.00 (0.18) 


arts assessment 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Gentilcore, J. (2002). The effect of an afterschool academic intervention service on a New York state eighth grade English language arts 
assessment: A case study (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3057258) 
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ACADEMIC PROGRAMS 


Academic Remediation Program (South Carolina) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This academic remediation program, offered in two rural Location 
middle schools in Orangeburg County, South Carolina, aimed LOCALE REGION 
to help struggling seventh grade African American students 
improve their scores on state standardized achievement tests. 
The program was taught by certified teachers and consisted 
of afterschool classes in each of the four core content areas 
(math, reading, science, and social studies). Students rotated 
through all four areas regardless of their areas of weakness. Racal Sactheast 
Teachers used test preparation manuals aligned to the state 
tests and provided worksheets from other sources. The 
program was offered for seven months, three days per week, 
for two hours per day. Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grade 7 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, a 0, 
ment, Other Achievement, Physical Activity/Health, Promotion & 80% Low-Income 100% Black 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Certified teachers 
DURATION: 7 months, 3 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Program manual 
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ACADEMIC PROGRAMS | ACADEMIC REMEDIATION PROGRAM (SOUTH CAROLINA) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impacts of attending an afterschool program offering remediation in English language 
arts, social studies, math, and science. Outcomes for students who participated were compared to outcomes for a matched 
group of non-participants. The comparison students experienced “business as usual,” which could have included other 
school- and community-sponsored afterschool activities. The study was conducted in 2005-06 in two rural middle schools 
in Orangeburg County, South Carolina. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ‘ rene Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Palmetto Achievement 


Challenge Test - co oo Grade 7 145 -0.05 (0.17) 
mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 
co Meets Tier Il standards for cause-and-effect. Does not meet ESSA Tier Il because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . p ater Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Palmetto Achievement 
Challenge Test - English co oo Grade 7 145 -0.23 (0.17) 


language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Patterson, P. W. (2007). The benefits of an afterschool program on African American students in two rural schools (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3361406) 
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ACADEMIC PROGRAMS 


After-School Academy 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


After-School Academy, offered in one middle school in Location 

Georgia during 2006-07, was intended to improve academic LOCALE REGION 
skills of low-achieving students. The program was staffed by 
certified math, science, and English/language arts teachers. 
Teachers were encouraged to follow the curriculum of the 
regular school day in their afterschool lessons. After-School 
Academy was offered for six months, three days per week, 
two hours each day. 


Not Reported Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 6-8 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, 0, 
ment, Other Achievement, Physical Activity/Health, Promotion & i ms cee at the id s coin 7 
Graduation, Attendance & Enrollment, School Engagement, Science Serr ery pee a een 0 ee 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: 6 months, 3 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | AFTERSCHOOL ACADEMY 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment the impact of an afterschool program in one middle school in Georgia. Outcomes for students who participated 
were compared to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which 
could have included other school- and community-sponsored afterschool activities. The study was conducted in 2005-06. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome F i rae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Georgia Criterion- 


Ref 
eferenced oS ©0 Grades 6-8 38 0.00 (0.32) 
Competency Test - 


mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome E e ean. Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Georgia Criterion-Ref- 
erenced Competency 8 8 Grades 6-8 38 -012 (0.32) 


Test - reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Ogden, C. (2008). Measuring the effectiveness of afterschool programs via participants’ pre and posttest performance levels on the Georgia 
Criterion Referenced Competency Test (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3297561) 
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ACADEMIC PROGRAMS 


Pitcsancsleteys Wels mialecce belen Tutoring Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Intervention Services Inc. tutoring program, offered in Location 


one alternative school in central Florida in 2009, aimed to LOCALE REGION 
increase the reading levels of struggling middle school readers 

in a short period of time. The program provided small-group 

instruction and used a variety of strategies, including differen- 

tiated instruction, personalized instruction, and response to i | 

intervention. The program was offered for six months, two days 


per week, 1.5 hours each day. 


Urban Southeast 
Sf od Om AYA ee We cd NV ea 
Participants 
CLASS GRADE TARGET STUDENTS 
No Effect 
Grades 6-8 Low-Achieving Students 


The study found neither improved nor negative outcomes for 


Attendance & Enrollment 


: SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
General Achievement 


Negative Effect 


The study found at least one negative outcome with no 
overriding contrary evidence for 


Reading/ELA Achievement 


9 a 0, 
NOT MEASURED FOR THIS INTERVENTION: Mathematics sei Low-neome ave hisek 
Achievement, Other Achievement, Physical Activity/Health, Promotion 16% White 
& Graduation, School Engagement, Science Achievement, and Social & 26% Latino 
Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 6 months, 2 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual 
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ACADEMIC PROGRAMS | INTERVENTION SERVICES INC. TUTORING PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool reading program offered by Intervention Services Inc. for students in 
grades 6-8 who had been expelled from their schools. Outcomes for students who participated were compared to outcomes for a 
matched group of non-participants. The comparison students experienced “business as usual,” which could have included other school- 
and community-sponsored afterschool activities. The study was conducted in 2009 in one school in central Florida. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school 
co co Grades 6-8 51 0.23 (0.28) 


days absent 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : 5 a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
GPA co co Grades 6-8 51 -0.50 (0.28) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; e mise Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 
Group Reading and 
ALON ee 00 00 Grades 6-8 51 -0.79 (0.29)" 

Diagnostic Evaluation 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III -0.79 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


CITATION: Lightner, P. A. (2010). Expelled middle school students: A study of the effects of a short-term, afterschool reading intervention program 
(Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3423819) 
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ACADEMIC PROGRAMS 


Language Workshop 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Language Workshop, offered in one middle school in Southern Location 
California in 2006-07, aimed to develop the academic vocabulary, LOCALE REGION 
word exposure, and word knowledge of English learners. 

The program involved a focus on target words, instruction on 

methods for independent vocabulary acquisition, and direct 

instruction related to word definitions. The program was 

offered for five weeks, four days per week, for 75 minutes 


Pedey, Suburban West 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 6-8 English Language Learners 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement Other Achievement 
. . 0 : 
Physical Activity/Health, Promotion & Graduation, Attendance & Not Reported 73% Latino 
Enrollment, School Engagement, Science Achievement, and 22% Asian 
Social & Emotional Competencies 5% Other 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Foreign language SETTING: School site 


BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group format 
STAFFING: Specialized staffing 
DURATION: 5 weeks, 4 days/week, 1.25 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | LANGUAGE WORKSHOP 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment, originally designed as a randomized controlled trial, estimates the impact of Language Workshop on vocabulary 
development for English learners in grades 6-8 in one middle school in Southern California. Outcomes for students who participated 
were compared to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which 
could have included other school- and community-sponsored afterschool activities. The study was conducted in 2006-07. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . 5 wee: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Peabody Picture 
Vocabulary Test-lll - oo co Grades 6-8 37 0.05 (0.33) 
Vocabulary Measures 


Test of Word Reading 
Efficiency (TOWRE) 
- Word Recognition 

measure 


oo oo Grades 6-8 37 0.26 (0.33) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Townsend, D. R. (2007). The academic vocabulary development of middle school English learners: An intervention study (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3273870) 
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ACADEMIC PROGRAMS 


New Jersey Test Prep Tutoring Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This New Jersey test prep tutoring program, offered in one Location 
middle school in a large district in 2006-07, aimed to improve LOCALE REGION 
the math or language arts standardized test scores of eighth 

grade students who had previously scored near-passing on 

either of these tests. The program, taught by certified teachers, 

offered small-group tutoring and instruction on test-taking i | 

strategies. The program was offered for six months, two days 


per week, 1.5 hours each day. 


Urban Northeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grade 8 Low-Achieving Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, = 0, 
ment, Other Achievement, Physical Activity/Health, Promotion & 60% Low-Income 40% Black 
Graduation, Attendance & Enrollment, School Engagement, Science 35% Latino 
Achievement, and Social & Emotional Competencies 25% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers 
DURATION: 6 months, 2 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual, 
ongoing support 
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ACADEMIC PROGRAMS | NEW JERSEY TEST PREP TUTORING PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool tutoring program for students who scored just below the passing score on 
the New Jersey state standardized mathematics test, English language arts test, or both tests. Outcomes for students who participated 
were compared to outcomes for a matched group of non-participants. The comparison students experienced “business as usual,” which 
could have included other school- and community-sponsored afterschool activities. The study was conducted in 2006-07. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


New Jersey Grade Eight 
Proficiency Assessment co co Grade 8 57 0.45 (0.28) 
- mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : ag Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


New Jersey Grade Eight 
Proficiency Assessment oo oo Grade 8 60 0.41 (0.27) 
- English language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: Rothman, T., & Henderson, M. (2011). Do school-based tutoring programs significantly improve student performance on standardized tests? 
RMLE Online: Research in Middle Level Education, 34(6), 1-10. 
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ACADEMIC PROGRAMS 


Academic Achievement Academy - Math Tutoring 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This program, offered in 20 schools in the Wake County (NC) Location 

Public School System in 2011-12, sought to increase student LOCALE REGION 

academic achievement among students close to proficient 

on End-of-Grade and End-of-Course mathematics exams. i 

The program provided small-group tutoring in mathematics, 

conducted by a certified teacher from the school. Tutoring | 

sessions were held for 11 weeks, two days per week, 1.5 hours 

cachiday, Suburban, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 4-8 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 


0, = 9 9 A 
Graduation, Attendance & Enrollment, Reading/ELA Achievement, 69% Low-Income 61% Black 1% Asian 
School Engagement, Science Achievement, and Social & Emotional 22% Latino 4% Other 
Competencies 13% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced, positive DELIVERY FORMAT: Group format 


adult-student relationships 
em " STAFFING: Certified teachers, specialized staffing 


DURATION: 10 weeks, 2 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Provider training 
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ACADEMIC PROGRAMS | ACADEMIC ACHIEVEMENT ACADEMY - MATH TUTORING 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool tutoring program for students who were close to proficient on End-of- 
Grade and End-of-Course exams. Outcomes for students who participated were compared to outcomes for a matched group of non- 
participants. The comparison students experienced “business as usual,” which could have included other school-and community- 
sponsored afterschool activities. The study was conducted in 20 schools in the Wake County Public School System in 2011-12. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome , r ear Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 

Score on End-of- 

Grade Mathematics oo oo Grades 4-8 658 -0.06 (0.07) 
Assessment 

Met mathematics 

oo oo Grades 4-8 658 0.07 (0.08) 


growth target 


Average effect size for Mathematics Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Rhea, A. (2013). Academic Achievement Academy (AAA), 2011-12 (D & A Report No. 13.03). Cary, NC: Wake County Public School System, 
Data and Accountability Department. 
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ACADEMIC PROGRAMS 


Academic Achievement Academy — Reading Tutoring 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This program, offered in 20 schools in the Wake County (NC) Location 
Public School System in 2011-12, sought to increase student LOCALE REGION 
academic achievement among students close to proficient on 

End-of-Grade and End-of-Course reading exams. The program Fy 


provided small-group tutoring in reading, conducted by a 


certified teacher from the school. Tutoring sessions were held | 
for 11 weeks, two days per week, 1.5 hours each day. 
Suburban, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 4-8 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Negative Effect 


The study found at least one negative outcome with no 
overriding contrary evidence for 

Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


ment, Mathematics Achievement, Other Achievement, Physical 


0, = 9 9 
Activity/Health, Promotion & Graduation, Attendance & Enrollment, 69% Low-Income 63% Black 1% Asian 
School Engagement, Science Achievement, and Social & Emotional 22% Latino 3% Other 
Competencies 11% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Sequenced, positive adult- DELIVERY FORMAT: Group format 


student relationships 
" ss STAFFING: Certified teachers, specialized staffing 


DURATION: 10 weeks, 2 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Provider training 
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ACADEMIC PROGRAMS | ACADEMIC ACHIEVEMENT ACADEMY —- READING TUTORING 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool tutoring program on reading outcomes for students who were close 

to proficient on End-of-Grade and End-of-Course exams. Outcomes for students who participated were compared to outcomes for a 
matched group of non-participants. The comparison students experienced “business as usual,” which could have included other school- 
and community-sponsored afterschool activities. The study was conducted in 2011-12. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . j eg: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Score on End-of-Grade 

: co oo Grades 4-8 978 -0.07 (0.06) 

Reading Assessment 

Met reading growth 
ad oo oo Grades 4-8 978 -0.18 (0.06)* 


target 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: Rhea, A. (2013). Academic Achievement Academy (AAA), 2011-12 (D & A Report No. 13.03). Cary, NC: Wake County Public School System, 
Data and Accountability Department. 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Louisville) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services, offered in Louisville, Location 
Kentucky during 2009-10, aimed to increase student achieve- LOCALE REGION 
ment in math and/or reading for students in low-performing 

Title | schools. The program provided tutoring, typically for 

several months in the second half of the school year. Length 

and frequency of sessions and approaches taken varied. i | 

Providers ranged from local community-based organizations to 


large national for-profit organizations. 


Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 4-8 General Population 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, a! 0, A 
ment, Other Achievement, Physical Activity/Health, Promotion & 100% Low-Income 30% White 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff 
DURATION: 3-5 months, 2 days/week, 1 hour/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (LOUISVILLE) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of Supplemental Educational Services for students in grades 4-8 in Jefferson County (KY) 
Public Schools. Outcomes for students who participated were compared to outcomes for a matched group of non-participants. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 
activities. The study was conducted during 2009-10. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i ; eos Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Kentucky Core Content 
“) 00 oo Math 656 0.06 (0.08) 
Test - mathematics tutoring 
Reading 
Kentucky Core Content 
Test eet = = eet eee oe C07) 
tutoring 


Average effect size for Mathematics Achievement outcomes at Tiers I-lll 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome z . Tes Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Kentucky Core Content Reading 
© 00 : vi .04 (0. 
Test - reading tutoring 68 0.04 (0.08) 
Reading 
Kentucky Core Content 
a Meee oo co & math 852 -0.02 (0.07) 
9 tutoring 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: Mufoz, M. A., Chang, F., & Ross, S. M. (2012). No Child Left Behind and tutoring in reading and mathematics: Impact of supplemental 
educational services on large scale assessment. Journal of Education for Students Placed at Risk, 17(3), 186-200. 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Milwaukee) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services offered in Milwaukee, Location 


Wisconsin from 2008-11 aimed to increase achievement in LOCALE REGION 
math and/or reading for students in low-performing Title | 

schools. The services assessed in this study were offered 

to students in grades 3-8 who were not proficient on state 

standardized tests. The program provided tutoring and other i | 

education interventions in math and/or language arts. The 


services were offered by the district and state-approved 


Urban Midwest 
private providers. The program was offered for the full school 
year, and the average participant received 26 hours of tutoring. 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 3-8 Low-Achieving Students 


No Effect 


The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mathematics Achievement 


Negative Effect 


The study found at least one negative outcome with no 
overriding contrary evidence for 


Reading/ELA Achievement 


87% Low-Income 68% Black 4% Asian 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 19% Latino 2% Other 
ment, Other Achievement, Physical Activity/Health, Promotion & : 
7% White 


Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help SETTING: School and community sites 
BEST PRACTICES USED: Positive adult- DELIVERY FORMAT: Group and one-to-one formats 


student relationships 
Peery STAFFING: Certified teachers, program staff 


DURATION: Full school year, 1 hour/day, 
average attendance=26 sessions 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (MILWAUKEE) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool tutoring programs on mathematics and reading outcomes for students in 
Milwaukee, Wisconsin. The comparison group is comprised of students eligible for afterschool tutoring who did not attend any tutoring 
sessions. The comparison students experienced “business as usual,” which could have included other school- and community-spon- 
sored afterschool activities. The study examined impacts for four years (2008-11). 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : 5 ve: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Wisconsin Knowledge 
and Concepts Evaluation 8 8 Year 1 4,772 -0.05 (0.03) 
— mathematics 
Wisconsin Knowledge 
and Concepts Evaluation 8 8 Year 2 1,870 -0.05 (0.05) 
— mathematics 
Wisconsin Knowledge 
and Concepts Evaluation 8 8 Year 3 2,830 -0.04 (0.04) 


— mathematics 


Wisconsin Knowledge 
and Concepts Evaluation 8 8 Year 4 3,662 0.03 (0.03) 


— mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl | eos 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (MILWAUKEE) 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; or Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Wisconsin Knowledge 
and Concepts 8 8 Year 1 4,696 -0.08 (0.03)* 
Evaluation — reading 
Wisconsin Knowledge 
and Concepts 8 8 Year 2 1,840 -0.08 (0.05) 
Evaluation — reading 
Wisconsin Knowledge 
and Concepts 8 8 Year 3 2,826 0.02 (0.04) 
Evaluation — reading 
Wisconsin Knowledge 


and Concept 8 8 Year 4 3,668 -0.02 (0.03) 
Evaluation — reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Heinrich, C. J., Burch, P., Good, A., Acosta, R., Cheng, H., Dillender, M., . . ., Stewart, M. (2014). Improving the implementation and 
effectiveness of out-of-school-time tutoring. Journal of Policy Analysis and Management, 33(2), 471-494. 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (Mississippi) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This Supplemental Educational Services program, offered in Location 

one rural elementary school in Mississippi, provided tutoring LOCALE REGION 

that aimed to improve ELA standardized achievement scores 

for elementary school students. Students who scored minimal 

or basic on state English language arts tests were invited 

to participate. No information is given on the content of the 

tutoring program. The program was offered for the full school 

year, three days per week, two hours each day. Reval Sactheast 
Participants 
CLASS GRADE TARGET STUDENTS 

EFFECTIVENESS AT A GLANCE Grades 3-6 Low-Achieving Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 

No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


0, a 0 
Physical Activity/Health, Promotion & Graduation, Attendance & 100% Low-Income 96% Black 
Enrollment, School Engagement, Science Achievement, and Social 3% Latino 
& Emotional Competencies 1% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Individualized, one-to-one format 


STAFFING: Not reported 
DURATION: Full school year, 3 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (MISSISSIPPI) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool tutoring program offered in one elementary school in rural Mississippi. 
Outcomes for students who participated were compared to outcomes for a matched group of non-participants. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 
The study was conducted in 2008-09. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . j eg: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Mississippi Curriculum 
ae § § Grades 3-6 169 0.11 (0.45) 


Test - language arts 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Lacy, S. S. (2011). The impact of afterschool tutoring on reading achievement of elementary students in a Mississippi rural school setting 
(Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3450309) 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (North Carolina) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services offered in North Carolina Location 
in 2007-08 aimed to increase achievement in math and/ LOCALE REGION 
or reading for students in low-performing Title | schools. 

Services were targeted to students in grades 3-8 who were not i 


proficient on state standardized tests. The program provided 


tutoring and other academic supports, from a range of provid- ) 
ers in math and/or language arts during non-school hours. The 
program was offered for the full school year, and the average Subuiban, Raxal Urban Sautheadt 
participant received 31 hours of tutoring. 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 3-6 Low-Achieving Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical 


0, = 0, 9 A 
Activity/Health, Promotion & Graduation, Attendance & Enrollment, 97% Low-Income 79% Black 1% Asian 
School Engagement, Science Achievement, and Social & Emotional 12% White 5% Other 
Competencies 4% Latino 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Not reported 
DURATION: Full school year, average 31 hours of programming 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (NORTH CAROLINA) 


Study Details 


STUDY DESCRIPTION 

To estimate the effects of supplemental services on outcomes for students in grades 3-8, this quasi-experimental study statistically 
matched students who received the intervention to students who were eligible for but did not receive services. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 
The study was conducted in North Carolina in 2007-08. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; w@: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
North Carolina End-of- 
co co Grades 3-8 3,942 -0.05 (0.03) 


Grade Test - reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: Gallagher, B., & Zoblotsky, T. (2009). Supp/emental educational services in the state of North Carolina: 2007-2008 student achievement 
analysis (CREP Report). Memphis, TN: Center for Research in Educational Policy, University of Memphis. 
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MULTI-STUDY PROGRAM SUMMARY 


Supplemental Educational Services (Virginia) 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


Supplemental Educational Services offered in 
Virginia in 2007-08 and 2008-09 aimed to increase 


student achievement in math and/or reading for 


No Effect 


Overall, the studies found neither improved nor 
negative outcomes for 


students in low-performing Title | schools. The ; , 
: Setter Mathematics Achievement 
services assessed in this study were offered to 


students in grades 3-8 who were not proficient on Reading/ELA Achievement 


state standardized tests. The program provided 
tutoring and other educational interventions in math 
and/or reading offered by state-approved providers. 


This review identified two studies of this program. 
The findings from each of these studies are presented individually on pages 114 — 117 of this evidence guide. 


- Ford, J., Harrison, L., Neergaard, L. Qian, H., Zoblotsky, T., McKay, D., & Ross, S. M. (2009). Supplemental 
educational services in the Commonwealth of Virginia: 2007-2008 (CREP Report). Memphis, TN: Center for 


Research in Educational Policy, University of Memphis. 


« McKay, D., Paek, J., Harrison, L., Qian, H., Zoblotsky, T., Ross, S. M., Fedde, F., & Ford, J. (2008). Supplemental 
education services in the state of Virginia: 2006-2007 (CREP Report). Memphis, TN: Center for Research in 


Educational Policy, University of Memphis. 
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Supplemental Educational Services (Virginia - 2006-07) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services offered in Virginia in Location 

2006-07 aimed to increase student achievement in math and/ LOCALE REGION 

or reading for students in low-performing Title | schools. The 

program was offered to students in grades 4-8 who were not i 

proficient on state standardized tests. The program consisted 

of tutoring and other supports in math and/or language arts (=) ) 

offered by state-approved providers. 
Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 4-8 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


100% Low-Income 91% Black 1% Asian 
4% White 2% Other 
2% Latino 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (VIRGINIA) | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

To estimate the effects of supplemental services on outcomes for students in grades 4-8, this quasi-experimental study statistically 
matched students who received the intervention to students who did not receive services. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study was conducted 
in Virginia in 2006-07. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . 5 acre Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Standards of Learning — 
: co co Grades 4-8 300 0.06 (0.12) 


mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; , ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Standards of Learning — 
Y co co Grades 4-8 540 -0.14 (0.09) 


reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: McKay, D., Paek, J., Harrison, L., Qian, H., Zoblotsky, T., Ross, S. M., Fedde, F., & Ford, J. (2008). Supp/emental educational services in the 
state of Virginia: 2006-2007 (CREP Report). Memphis, TN: Center for Research in Educational Policy, University of Memphis. 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (VIRGINIA) | SUMMARY #2 


Supplemental Educational Services (Virginia - 2007-08) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services offered in Virginia in Location 
2007-08 aimed to increase student achievement in math and/ LOCALE REGION 
or reading for students in low-performing Title | schools. The 
services assessed in this study were offered to students in i 
grades 4-8 who were not proficient on state standardized tests. 
The program consisted of tutoring and other supports in math ) 
and/or language arts offered by state-approved providers. 
Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 4-8 Low-Achieving Students 
No Effect SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, 2! 
ment, Other Achievement, Physical Activity/Health, Promotion & 100% Low-Income Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Not reported 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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MULTI-STUDY PROGRAM SUMMARY SUPPLEMENTAL EDUCATIONAL SERVICES (VIRGINIA) | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

To estimate the effects of supplemental services on outcomes for students in grades 4-8, this quasi-experimental study statistically 
matched students who received the intervention to students who did not receive services. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study was conducted 
in Virginia in 2007-08. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . 5 acre Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Standards of Learning — 
ce 0 co Grades 4-7 606 0.07 (0.08) 
mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 0.07 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; , ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Standards of Learning — 
: co co Grades 4-7 716 -0.09 (0.07) 


reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-Ill 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: Ford, J., Harrison, L., Neergaard, L. Qian, H., Zoblotsky, T., McKay, D., & Ross, S. M. (2009). Supplemental educational services in the 
Commonwealth of Virginia: 2007-2008 (CREP Report). Memphis, TN: Center for Research in Educational Policy, University of Memphis. 
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ACADEMIC PROGRAMS 


College Possible 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


College Possible, offered in five high schools in Minneapolis Location 
and St. Paul during 2010-12, aimed to increase college LOCALE REGION 
enrollment and persistence for low-income students in grades 

11 and 12. The two-year afterschool program provided ACT/SAT 

preparation, including intensive tutoring, and college counsel- 

ing regarding admissions, financial aid, and the transition to H | 


college. The program offered 320 hours of programming. 


Urban Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grade 11 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 
General Achievement 
Mathematics Achievement 
Reading/ELA Achievement 


Attendance & Enrollment 


100% from families 9% White 
with incomes below 91% Other 
NOT MEASURED FOR THIS INTERVENTION: Physical Activity/ the local median 


Health, Other Achievement, Promotion & Graduation, School Engage- 
ment, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, college counseling SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Two school years, 320 hours of programming 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | COLLEGE POSSIBLE 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact on academic achievement and college application of College Possible for students 
in five high schools in Minneapolis and St. Paul. Outcomes for students randomized to the program are compared to outcomes for 
students in the comparison group. The comparison students experienced “business as usual,” which could have included other school- 
and community-sponsored afterschool activities. The study was conducted in 2010-12. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome e Money Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
ACT composite ) ) Grade 11 171 -0.02 (0.15) 


Average effect size for General Achievement outcomes at Tiers I-IIl 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 

Outcome : F Ray: Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 

ACT mathematics 
) ) Grade 11 171 0.01 (0.15) 
subtest 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 0.01 


Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
O° G Number of Effect size 
assis): Without site and With site and fone students (Standard error) 
sample criteria sample criteria 
ACT English subtest > ) Grade 11 171 0.05 (0.15) 
ACT reading subtest > ) Grade 11 171 -0.06 (0.15) 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III -0.01 


Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 
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ACADEMIC PROGRAMS | COLLEGE POSSIBLE 


Study Details (Continued) 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome - e og Group 
Without site & sample With site & students (Standard error) 
criteria sample criteria 
ACT science subtest > y) Grade 11 171 -0.05 (0.15) 
Average effect size for Science Achievement outcomes at Tiers I-lll -0.05 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Avery, C. (2013). Evaluation of the College Possible program: Results from a randomized controlled trial (NBER Working Paper No. 19562). 
Cambridge, MA: National Bureau of Economic Research. 
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ACADEMIC PROGRAMS 


Texas Afterschool Math Remediation 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Texas Afterschool Math Remediation Program, offered in Location 
one Texas school district during 1999-2002, aimed to increase LOCALE REGION 
the mathematics achievement of at-risk ninth grade students. 

Certified teachers offered individual tutoring, and the program 

included small peer-study groups focused on specific con- 

cepts. The program used the math component of LightSpan i | 

Achieve Now software, which offers interactive math games 


through Sony PlayStation. Students could take PlayStations 


Urban Southwest 
home to continue the math games. The program was offered 
four days per week for 1.5 hours each day. 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grade 9 Low-Achieving Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & Not Reported 65% Latino 9% Black 


Graduation, Reading/ELA Achievement, Attendance & Enrollment, 
School Engagement, Science Achievement, and Social & Emotional 25% White 1% Other 
Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Active learning DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers 
DURATION: 4 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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ACADEMIC PROGRAMS | TEXAS AFTERSCHOOL MATH REMEDIATION 


Study Details 


STUDY DESCRIPTION 

This study uses a quasi-experimental design to estimate the impact of a voluntary math remediation afterschool program for ninth 
graders on scores on an Algebra | end-of-course exam and the Algebra | course grade. The study, conducted in a Texas school district, 
compares outcomes for students who participated in the program during 1999-2002 to a matched sample of at-risk students who 

did not attend the program. The comparison students experienced “business as usual,” which could have included other school- and 


community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; aon Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Algebra 1 
2 § § Grade 9 287 -0.02 (0:7) 
course grade 
Algebra 1 end 
9 § § Grade 9 279 0.16 (0.18) 
of course exam 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 0.07 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Carroll, S. (2004). The effect of after school tutoring and summer enrichment programs on the algebra I achievement of at-risk students 
(Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3122343) 
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ACADEMIC PROGRAMS 


Supplemental Educational Program (Texas) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This program, offered in one urban school district in Texas Location 


in 2006-07, aimed to increase the academic achievement LOCALE REGION 
and attendance of low-income, at-risk urban ninth graders. 

The school-sponsored program provided instruction in social 

studies as a vehicle through which to address other skills. The 

program was led by teachers and volunteer tutors from a local i | 

university and drew on the Model Lessons Curriculum and 


exercises from Texas state standardized tests. Activities were 


Urban Southwest 

hands-on and organized around problem solving. The program 
was offered for 12 weeks, two days per week, for one hour 
each day. 

Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE 

Grade 9 Low-Income, 
cas Low-Achieving 

No Effect 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 
The study found neither improved nor negative outcomes for 


Attendance & Enrollment 


Negative Effect 


The study found at least one negative outcome with no 
overriding contrary evidence for 


Other Achievement 
Not Reported 48% Black 8% White 
36% Latino 8% Asian 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Physical Activity/Health, Promotion & 
Graduation, Reading/ELA Achievement, School Engagement, Science 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Active learning DELIVERY FORMAT: Not reported 


STAFFING: Certified teachers, volunteers 
DURATION: 12 weeks, 2 days/week, 1 hour/day 


IMPLEMENTATION SUPPORT: Program manual, provider training 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL PROGRAM (TEXAS) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of supplemental educational services for ninth graders in one urban school district in Texas. 
Program participants scored below 70% in reading and social studies for two consecutive six-week periods and had unexcused school 
lateness and/or absences. Outcomes for participants are compared to outcomes for a matched group of students who did not partici- 
pate. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. The study was conducted during 2006-07. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome E 5 ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school 
g g Grade 9 64 0.11 (0.25) 
days absent 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 0.11 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


OTHER ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; aoe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Evaluator-designed 
measure of social 
§ § Grade 9 64 -0.63 (0.26)* 


studies knowledge 
and skills 


Average effect size for Other Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


CITATION: Felix, E. E. (2009). The impact of a supplemental education program on academic performance and attendance of ninth grade students at 
risk for school failure (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3351551) 
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ACADEMIC PROGRAMS 


Supplemental Educational Services (New Mexico) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Supplemental Educational Services offered in New Mexico Location 
in 2006-07 aimed to increase achievement in math and/or LOCALE REGION 
reading for students in low-performing Title | schools. The 
program consisted of afterschool tutoring and other academic i 
supports in math and/or language arts. Services were offered 
by 18 state-approved providers, including for-profit and non- (=) B 
profit organizations, that varied in the length of their sessions 
and the credentials of their tutors. Subuiban, Raxal Urban Grstieieat 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE 
Not Reported Low-Achieving Students 


ive Eff 
Negat ig ect SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


The study found at least one negative outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 
100% Low-Income 73% Latino 1% Black 
5% White 19% Other 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Certified teachers, program staff 
DURATION: Full school year, 1-4 days/week, 1-4 hours/day 


IMPLEMENTATION SUPPORT: Ongoing support 
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ACADEMIC PROGRAMS | SUPPLEMENTAL EDUCATIONAL SERVICES (NEW MEXICO) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on mathematics and reading achievement of supplemental educational services for 
students in New Mexico. Outcomes for students who participated were compared to outcomes for a matched group of non- 
participants. The comparison students experienced “business as usual,” which could have included other school-and community- 
sponsored afterschool activities. The study was conducted in 2006-07. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 

Outcome i ; ae Group 

Without site and With site and students (Standard error) 
sample criteria sample criteria 
New Mexico Standards Not 
(e) 

Based Assessment - ol -0.2 .03)* 
§ § Reported 3,310 0.26 (0.03) 


mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 

Outcome . ; WEG: Group 

Without site and With site and students (Standard error) 
sample criteria sample criteria 
New Mexico Standards Not 
fo) 
Based Assessment - FE y 

§ § Reported 3,986 0.06 (0.03) 


reading 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Marquez, S. A., & Kovacic, C. B. (2007). Supplemental educational services evaluation report, 2006-2007 (Unpublished report). Santa Fe, 
NM: New Mexico Public Education Department. 
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Afterschool Plus Social Supports Programs 


MULTI-STUDY PROGRAM SUMMARY 


Challenging Horizons Program 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


The Challenging Horizons Program aimed to No Effect 
support middle school children diagnosed with Overall, the studies found neither improved nor 
ADHD and improve their behavior, social skills, negative outcomes for 


and academic outcomes. The program included pes 

eet at : ; ~ General Achievement 
regular individual meetings with a counselor to ‘ 
set and monitor behavioral and academic goals, a _ Promotion & Graduation 


group intervention focused on addressing social ©) Social & Emotional Competencies 
impairment, recreational time, education/study skills : 
time, and individual homework help time. 


‘This review identified two studies of this program. The findings from each of these studies are presented 
individually on pages 129 — 133 of this evidence guide. 


- Evans, S. W., Langberg, J. M., Schultz, B. K., Vaughn, A., Altaye, M., Marshall, S. A., & Zoromski, A. K. (2016). 
Evaluation of a school-based treatment program for young adolescents with ADHD. Journal of Consulting and 
Clinical Psychology, 84, 15-30. 


- Molina, B.S. G., Flory, K., Bukstein, O. G., Greiner, A. R., Baker, J. L., Krug, V., & Evans, S. W. (2008). Feasibility and 
preliminary efficacy of an afterschool program for middle schoolers with ADHD: A randomized trial in a large public 
middle school. Journal of Attention Disorders, 12(3), 207-217. 


MULTI-STUDY PROGRAM SUMMARY 


CHALLENGING HORIZONS PROGRAM | SUMMARY #1 


Challenging Horizons Program (Nine Sites) 


The Challenging Horizons Program, offered in nine middle 
schools in the Midwest and Southeast, aimed to support 
middle school children diagnosed with ADHD and improve 
their behavior, social skills, and academic outcomes. The 
program included regular individual meetings with a counselor 
to set and monitor behavioral and academic goals, a group 
intervention focused on addressing social impairment, 
recreational time, education/study skills time, and individual 
homework help time. The program was offered for the full 
school year, two days per week, 2.25 hours per day. The 
implementation year is not reported. 


Location 


LOCALE REGION 


Suburban, Rural, Urban Midwest & Southeast 


Participants 


CLASS GRADE TARGET STUDENTS 


No Effect 


The study found neither improved nor negative outcomes for 
"> General Achievement 


— Promotion & Graduation 


-) Social & Emotional Competencies 


NOT MEASURED FOR THIS INTERVENTION: Mathematics 
Achievement, Other Achievement, Physical Activity/Health, Reading/ 
ELA Achievement, Attendance & Enrollment, School Engagement, and 
Science Achievement 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Not Reported 83% White 


8% Black 


3% Latino 
21% Other 


Program Content 
MAIN COMPONENTS: Social skills activities, study skills 


BEST PRACTICES USED: Sequenced, active learning, 
positive adult-student relationships, family engagement 


Logistics 

SETTING: School site 

DELIVERY FORMAT: Group and one-to-one formats 
STAFFING: Specialized staffing, volunteers 

DURATION: Full school year, 2 days/week, 2.25 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, ongoing support 
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MULTI-STUDY PROGRAM SUMMARY CHALLENGING HORIZONS PROGRAM | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

The randomized controlled trial estimates the impact of the Challenging Horizons Program for students in grades 6-8 diagnosed with 
attention deficit hyperactivity disorder. The study was conducted in nine middle schools. Within school and medication status, students 
were randomly assigned to participate in the Challenging Horizons Program or to experience regular care. The implementation year is 
not reported. 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Classroom Performance 
Survey - academic 
subscale (Brady, Evans, ) » Grades 6-8 216 0.05 (0.14) 
Berlin, Bunford, & Kern, 
2012) 
Classroom Performance 
Survey - academic 
¥ , , Grades 6-8 216 -0.05 (0.14) 


subscale — six month 
follow up 


Average effect size for Promotion & Graduation outcomes at Tiers I-lll 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY CHALLENGING HORIZONS PROGRAM | SUMMARY #1 


SOCIAL & EMOTIONAL COMPETENCIES 


ESSA evidence tier 
Bice Group Number of Effect size 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Classroom Performance 
Survey - interpersonal 
subscale (Brady, ) ) Grades 6-8 216 0.03 (0.14) 
Evans, Berlin, Bunford, 
& Kern, 2012) 


Classroom Performance 
Survey - interpersonal 
subscale — six-month 

follow-up 


) 9 Grades 6-8 216 -0.09 (0.14) 


Social Skills 
Improvement System - 
social skills domain ) ) Grades 6-8 216 0.02 (0.14) 
(Gresham, Elliott, Vance, 
& Cook, 2011) 


Social Skills 
Improvement System - 
social skills domain — 

six-month follow-up 


9 9 Grades 6-8 216 0.06 (0.14) 


Average effect size for Social & Emotional Competencies outcomes at Tiers I-III 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Evans, S. W., Langberg, J. M., Schultz, B. K., Vaughn, A., Altaye, M., Marshall, S. A., & Zoromski, A. K. (2016). Evaluation of a school-based 
treatment program for young adolescents with ADHD. Journal of Consulting and Clinical Psychology, 84, 15-30. 
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MULTI-STUDY PROGRAM SUMMARY 


CHALLENGING HORIZONS PROGRAM | SUMMARY #2 


Challenging Horizons Program - Northeast 


The Challenging Horizons program, offered in a large public 
middle school near Pittsburgh, Pennsylvania in 2003, aimed 
to support middle school children diagnosed with ADHD and 
improve their behavior, social skills, and academic outcomes. 
The program included regular individual meetings with a 
counselor to set and monitor behavioral and academic goals, 
a group intervention focused on addressing social impairment, 
recreational time, education/study skills time, and individual 
homework help time. 


Location 


LOCALE REGION 


Northeast 


Participants 


CLASS GRADE TARGET STUDENTS 


No Effect 


The study found neither improved nor negative outcomes for 


\. General Achievement 


NOT MEASURED FOR THIS INTERVENTION: Mathematics 
Achievement, Other Achievement, Physical Activity/Health, Promotion 
& Graduation, Reading/ELA Achievement, Attendance & Enrollment, 
School Engagement, Science Achievement, and Social & Emotional 
Competencies 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Not Reported 


50% White 
48% Black 


Program Content 


MAIN COMPONENTS: Homework help, social skills, 
study skills 


BEST PRACTICES USED: Sequenced, active learning, 
family engagement, positive adult-student relationships 


Logistics 

SETTING: Not reported 

DELIVERY FORMAT: Group and one-to-one formats 
STAFFING: Specialized staffing 

DURATION: 10 weeks, 2 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY CHALLENGING HORIZONS PROGRAM | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 
This randomized controlled trial estimates the impact of the Challenging Horizons Program for students in grades 6-8 who were, after 


clinical evaluations, deemed eligible based on their diagnoses of attention deficit hyperactivity disorder and IQs of at least 70. Students 
were randomly assigned to the treatment or to experience regular care using stratified random assignment by gender and grade. The 
study was conducted in a large public middle school near Pittsburgh, Pennsylvania during 2003. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; . nae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Grade point average > > Grades 6-8 20 0.39 (0.45) 
Percentage of course 
2 , , Grades 6-8 20 0.70 (0.46) 
grades C or better 
Percentage of course 
=e , , Grades 6-8 20 0.61 (0.46) 
grades D or better 
Average effect size for General Achievement outcomes at Tiers I-lll 0.57 


( Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Molina, B.S. G., Flory, K., Bukstein, O. G., Greiner, A. R., Baker, J. L., Krug, V., & Evans, S. W. (2008). Feasibility and preliminary efficacy of an 
afterschool program for middle schoolers with ADHD: A randomized trial in a large public middle school. Journal of Attention Disorders, 12(3), 207-217. 
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AFTERSCHOOL PLUS SOCIAL SUPPORTS PROGRAMS 


Quantum Opportunity Program 


The Quantum Opportunity Program, offered in seven cities Location 
during 1995-2001, aimed to promote high school graduation LOCALE REGION 
and postsecondary enrollment among at-risk high school 
students attending schools with high dropout rates. The 
program offered case management, financial incentives, and 
other supportive services that followed students year-round for 


four years. Afterschool activities included one-on-one tutoring, 


computer-based instruction, college visits, life skills training, Wesk. Midwest. Base Soudieast 
> > > 


work-readiness training, and community service. 


Participants 
CLASS GRADE TARGET STUDENTS 


No Effect SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


The study found neither improved nor negative outcomes for 
General Achievement 
~ Mathematics Achievement 


~ Promotion & Graduation 


©) Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: Other Achievement, 


R 9 
Physical Activity/Health, Attendance & Enrollment, School Engagement, met epontest 66% Black 
Science Achievement, and Social & Emotional Competencies 25% Latino 
Program Content Logistics 
MAIN COMPONENTS: Community service, drug/alcohol SETTING: School and community sites 


awareness, financial incentives, mentoring, tutoring, social 
“ apiece a banaue DELIVERY FORMAT: Group and one-to-one formats 
skills, study skills 


STAFFING: Program staff, specialized staffing 
BEST PRACTICES USED: Sequenced, active learning, 


family engagement, positive adult-student relationships, DURATION: Four years, over 750 hours/year offered 


staff-student cultural similarity IMPLEMENTATION SUPPORT: Implementation monitoring 
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AFTERSCHOOL PLUS SOCIAL SUPPORTS PROGRAMS | QUANTUM OPPORTUNITY PROGRAM 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact on high school graduation of the Quantum Opportunity Program for high school 
students in seven cities (Cleveland, Fort Worth, Houston, Memphis, Philadelphia, Washington, DC, and Yakima). Ninth graders with an 
eighth grade GPA in the lower two-thirds of the distribution in their school cohort were randomly assigned to participate in the program 
or to the comparison group. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. The study was conducted during 1995-2001. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; Too Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
° Cannot 
Grade Point Average 5 ; 
g ) ) Grades 9-12 1,069 dctarmine 


Cannot 


Average effect size for General Achievement outcomes at Tiers I-lll _ 
determine 


( Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; “ eer Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
State assessments - Cannot 
‘ 0) > Grades 9-12 1,069 : 
mathematics determine 


Cannot 


Average effect size for Mathematics Achievement outcomes at Tiers I-III Ml 
determine 


( Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome E ; rin Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
High schools 
ie ) ) Grades 9-12 766 0.05 (0.07) 
credits earned 
High school 
= : ) » Grades 9-12 1,069 0.10 (0.06) 
graduation 
Average effect size for Promotion & Graduation outcomes at Tiers I-lll 0.08 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 
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AFTERSCHOOL PLUS SOCIAL SUPPORTS PROGRAMS | QUANTUM OPPORTUNITY PROGRAM 


READING ACHIEVEMENT 


State assessments — Cannot 
reading/ELA : : Graden a2 1,069 determine 


; : : : nn 
Average effect size for Reading Achievement outcomes at Tiers I-III Cannet 


determine 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATIONS: Schirm, A., Stuart, E., McKie, A. (2006). The Quantum Opportunity Program demonstration: Final impacts (MRP Report No. 8279-932). 
Washington, DC: Mathematica Policy Research. 


Schirm, A., Rodriguez-Planas, N., Maxfield, M., & Tuttle, C. (2003). The Quantum Opportunity Program demonstration: Short-term impacts (MPR Report 
No. 8279-093). Washington, DC: Mathematica Policy Research. 


Maxfield, M., Schirm, A., & Rodriguez-Planas, N. (2003). The Quantum Opportunity Program demonstration: Implementation and short-term impacts (MPR 
Report No. 8279-093). Washington, DC: Mathematica Policy Research. 


Maxfield, M., Castner, L., Maralani, V., & Vencill, M. (2003). The Quantum Opportunity Program demonstration: Implementation findings (MPR Report No. 
8279-080). Washington, DC: Mathematica Policy Research. 
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Arts Programs 


ARTS PROGRAMS 


Theater Arts Afterschool Program 


PROGRAM DESCRIPTION 


This theater arts afterschool program, offered in Los Angeles, 
aimed to increase the school engagement and academic 
performance of underachieving, gifted Latino middle school 
students. Students created and performed a theatrical adapta- 
tion of Shakespeare’s Romeo and Juliet. Youth were involved 
in all aspects of the production including writing, directing, 
acting, and producing the performance. The implementation 
year is not reported. 


EFFECTIVENESS AT A GLANCE 


No Effect 

The study found neither improved nor negative outcomes for 
Other Achievement 
Reading/ELA Achievement 


Science Achievement 


Negative Effects 


The study found at least one negative outcome with no 
overriding contrary evidence for 


School Engagement 


NOT MEASURED FOR THIS INTERVENTION: Attendance & Enroll- 


ment, General Achievement, Mathematics Achievement, Physical Activity/ 


Health, Promotion & Graduation, and Social & Emotional Competencies 


STUDY LOCATION AND PARTICIPANTS 


Location 


LOCALE REGION 


West 
Participants 
CLASS GRADE TARGET STUDENTS 
Underachieving, Gifted 
Grades 6-8 Latino Students 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Not Reported 100% Latino 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content 
MAIN COMPONENTS: Drama, theater 


BEST PRACTICES USED: Sequenced, active learning 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 


Logistics 

SETTING: School site 

DELIVERY FORMAT: Group format 
STAFFING: Certified teachers 
DURATION: 10 week program 


IMPLEMENTATION SUPPORT: Ongoing support 
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ARTS PROGRAMS | THEATER ARTS AFTERSCHOOL PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment compared grades and teacher-reported work habits for a group of gifted, underachieving, Latino middle school 
students who participated in a theater arts program to outcomes for non-participants. The comparison students experienced “business 
as usual,” which could have included other school and community-sponsored afterschool activities. The study was conducted in Los 
Angeles, California. The implementation year is not reported. 


OTHER ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . 5 acre Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
History course grade oo oo Grades 6-8 34 -0.45 (0.35) 


Average effect size for Other Achievement outcomes at Tiers I-lll 
co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : Tee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
English course grade § § Grades 6-8 34 Cannot 
determine 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III aaNet 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . x Renew Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Science course grade 8 8 Grades 6-8 34 -0.68 (0.35) 
Average effect size for Science Achievement outcomes at Tiers I-Ill -0.68 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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SCHOOL ENGAGEMENT 


School Attitude 
Assessment Survey - 
Revised - Attitudes 8 8 Grades 6-8 34 0.52 (0.35) 
Towards School 
subscale 


School Attitude 
Assessment Survey - 
Revised - Attitudes oo oo Grades 6-8 34 0.55 (0.35) 
Towards Teachers 
subscale 


School Attitude 
Assessment Survey - 


pevigcae Gaal 8 8 Grades 6-8 34 0.26 (0.35) 
Valuation subscale 
School Attitude 
Assessment Survey - 
Revised - Motivation/ oo) co Grades 6-8 34 0.46 (0.35) 
Self-Regulation 
subscale 
Teacher-reported work Cannot 
habits - English 8 3 Grades 2 a determine 
Teacher- k 
ge ee ea § § Grades 6-8 34 -0.89 (0.36)" 
habits - history 
Teacher-reported work Cannot 
habits - mathematics 8 8 Grades 6-8 a determine 
Teacher- k 
sic ie eal al § § Grades 6-8 34 0.31 (0.35) 


habits - science 


Average effect size for School Engagement outcomes at Tiers I-lll | oz 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Niederdeppe, N. N. (2009). The impact of a theatre arts intervention on underachieving gifted Latinos (Doctoral dissertation). Retrieved from 
ProQuest Dissertations and Theses Database (UMI No. 3351755) 
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Career and Leadership Programs 


MULTI-STUDY PROGRAM SUMMARY 


Citizen Schools 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


Citizen Schools is an afterschool program for Positive Effect 


low-income, academically struggling middle school Overall, the studies found at least one improved 


students. The program involves apprenticeships outcome with no overriding contrary evidence for 


and academic support, including structured Attendance & Enrollment 


homework help, tutoring support, and academic 

enrichment. Apprenticeships are the focus of the Morte inalics Sehicvemen 

program: students work with adult volunteers to Promotion & Graduation 

learn a skill or content area and present what they Reading/ELA Achievement 

learn in a public showcase. 
Mixed Effects 
The studies found both improved outcomes and 
null or negative outcomes 


School Engagement 


‘This review identified two studies of this program. The findings from each of these studies are presented 
individually on pages 143 — 148 of this evidence guide. 


- Acaira, E., Vile, J., & Reisner, E. R. (2010). Citizen Schools: Achieving high school graduation: Citizen Schools’ youth 


outcomes in Boston. Boston, MA: Policy Studies Associates, Inc. 


- Fountain, A. R., Gamse, B., Velez, M., Hillard, M., & Cropper, P. (2016). Evaluation of Citizen Schools’ expanded 
learning time model: Final report. Cambridge, MA: Abt Associates. 


MULTI-STUDY PROGRAM SUMMARY CITIZEN SCHOOLS | SUMMARY #1 


Citizen Schools (8th Grade Academy) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This Citizen Schools program, offered in Boston in 2001-06, Location 


was targeted to low-income, academically struggling middle LOCALE REGION 
school students. The program involved apprenticeships and 

academic support, including structured homework help, tutor- 

ing support, and academic enrichment. The Citizen Schools 8th 

Grade Academy also offered support for students and families | | i 


in the high school application process, visits to colleges and 


job sites, and opportunities to develop leadership and personal ian Nordieact 


decision-making skills. This was a full-year program. 


EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 
Positive Effect Grade 


The study found at least one improved outcome with no 


Oem NeIG conti ahy wie mea io SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Attendance & Enrollment 


Mathematics Achievement 
Promotion & Graduation 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- Not Reparted ine Repoxcell 
ment, Other Achievement, Physical Activity/Health, School Engage- 

ment, Science Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Apprenticeships, tutoring, SETTING: School site 


h k help, social skill 
EN ee seer DELIVERY FORMAT: Group and one-to-one formats 


BEST PRACTICES USED: Sequenced, active learning, 


STAFFING: Certified teachers, program staff, volunteers 
positive adult-student relationships, family engagement 


DURATION: Full school year, no other info provided 


IMPLEMENTATION SUPPORT: Provider training, 
implementation monitoring 
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MULTI-STUDY PROGRAM SUMMARY CITIZEN SCHOOLS | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the Citizen Schools 8th Grade Academy. High school outcomes for students who 
participated in the program as eighth graders during 2001-06 are compared to outcomes for a matched set of non-participants. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 


activities. The study was conducted in 2001-08. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 

Outcome ; ; ee Group 

Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percentage of school 
days attended, Tier Ill Tier Ill Grade 8 350 0.24 (0.1)* 
participants 


9th grade 


Average effect size for Attendance & Enrollment outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; Laas Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Earned credit in 9th ; , Grade 8 
Tier Ill Tier Ill ae 21 BP 14)* 
grade mathematics a participants 6 0.25 (0.14) 
Passing score on 
Massachusetts Grade 8 
Comprehensive § § aCe 202 0.16 (0.14) 


articipants 
Assessment System P P 


(MCAS) - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


* Statistically significant at p<.05 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY 


CITIZEN SCHOOLS | SUMMARY #1 


PROMOTION & GRADUATION 


Outcome 


Without site and 
sample criteria 


On-time promotion : 
to 10th grade vee 


Four-year 
Ms Tier Ill 


ESSA evidence tier 
NAT Group 
With site and 
sample criteria 
; Grade 8 
nigel participants 
: Grade 8 
Teel participants 


graduation rate 


Number of 
students 


404 


116 


Average effect size for Promotion & Graduation outcomes at Tiers I-lll 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


Outcome 


Without site and 
sample criteria 


Earned credit in 9th 


Tier Ill 
grade English ‘i 
Passing score on 
Massachusetts 
Comprehensive § 


ESSA evidence tier 
ee Group 
With site and 
sample criteria 
: Grade 8 
her participants 
Grade 8 
8 ee 
participants 


Assessment System 
(MCAS) — English 
language arts 


Number of 
students 


202 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


Effect size 
(Standard error) 


0.26 (0.10)* 


0.32 (0.19)* 


Effect size 
(Standard error) 


0.42 (0.14)* 


0.23 (0.14) 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Acaira, E., Vile, J., & Reisner, E. R. (2010). Citizen Schools: Achieving high school graduation: Citizen Schools’ youth outcomes in Boston. 


Boston, MA: Policy Studies Associates, Inc. 
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MULTI-STUDY PROGRAM SUMMARY CITIZEN SCHOOLS | SUMMARY #2 


Citizen Schools (35 Schools) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This Citizen Schools program, offered at sites in seven states Location 


during 2010-14, was targeted to low-income, academically LOCALE REGION 
struggling middle school students. The program involved 
apprenticeships and academic support, including structured 
homework help, tutoring support, and academic enrichment. 
Apprenticeships were the focus of the program: students 


worked with adult volunteers to learn a skill or content area 


and presented what they learned in a public showcase. The Not Reported All Regions 


program ran for the full school year, four days per week, 2.5 
hours each day. 


Participants 


en VA oe Wed eV red = CLASS GRADE TARGET STUDENTS 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mixed Effects 


The study found a mix of improved outcomes and null or 


negative outcomes for 
School Engagement 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


ment, Other Achievement, Physical Activity/Health, Promotion & Not Reported Not Reported 
Graduation, Attendance & Enrollment, Science Achievement, and 

Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Apprenticeships, tutoring, SETTING: School site 


h k hel 
ee DELIVERY FORMAT: Group and one-to-one formats 


BEST PRACTICES USED: Sequenced, active learning, 


STAFFING: Certified teachers, program staff, volunteers 
positive adult-student relationships, family engagement 


DURATION: Full school year, 4 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, ongoing support 
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MULTI-STUDY PROGRAM SUMMARY CITIZEN SCHOOLS | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on academic achievement and school engagement of Citizen Schools for students in 
grades 5-8. Outcomes for program participants are compared to outcomes for non-participants. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study was conducted 
at sites in seven states in 2010-14. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; eee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
State standardized 
Cannot 


assessments - Tier Ill Tier Ill Grades 5-8 2011-12 


' determine* 
mathematics 


- , , : Cannot 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


determine 


* Statistically significant at p<.05 
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MULTI-STUDY PROGRAM SUMMARY 


SCHOOL ENGAGEMENT 


Outcome 


Behavioral engagement 
of peers 


Behavioral engagement 
of peers 


Behavioral misconduct 
(self report) 


Behavioral misconduct 
(self report) 


Perception that teachers 
care about students 


Perception that teachers 
care about students 


Social engagement 


Social engagement 


* Statistically significant at p<.05 


ESSA evidence tier 


Without site and 
sample criteria 


Tier Ill 


Average effect size for School Engagement outcomes at Tiers I-lll 


With site and 
sample criteria 


Tier Ill 


CITIZEN SCHOOLS | SUMMARY #2 


Group 


2011-12 


2012-13 


2011-12 


2012-13 


2011-12 


2012-13 


2011-12 


2012-13 


Number of 
students 


299 


1004 


299 


1004 


299 


1004 


299 


1004 


Effect size 
(Standard error) 


Cannot 
determine* 


Cannot 
determine 


Cannot 
determine 


Cannot 
determine 


Cannot 
determine 


Cannot 
determine 


Cannot 
determine 


Cannot 
determine 


Cannot 


determine 


§ Meets Tier IIl standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Fountain, A. R., Gamse, B., Velez, M., Hillard, M., & Cropper, P. (2016). Evaluation of Citizen Schools’ expanded learning time model: Final 


report. Cambridge, MA: Abt Associates. 
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CAREER & LEADERSHIP PROGRAMS 


Stay-in-School for College and Career Opportunities (SISCO) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Stay-in-School for College and Career Opportunities (SISCO), Location 
offered in one school district in 2011-13, aimed to promote LOCALE REGION 
academic success, social and emotional learning, and college 

and career readiness for low-income Latino middle school 

students. Program activities included tutoring and academic 

enrichment in literacy, math, and science. In addition, the | | ij 


program offered career exploration activities, high school visits, 


parent-student workshops, and summer programming. The 


Urban Midwest 
program addressed social and emotional learning through a 
curriculum developed by ASPIRA, a national Latino educational 
organization. The program was offered for ten weeks, two days 
per week, one hour each day. Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 
Attendance & Enrollment 
_ Promotion & Graduation 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


: : ; : - Not Reported 100% Latino 
ment, Mathematics Achievement, Other Achievement, Physical Activity/ 
Health, Reading/ELA Achievement, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Tutoring SETTING: School site 
BEST PRACTICES USED: Family engagement, respect DELIVERY FORMAT: Group and one-to-one formats 


for stud nb caltures STAFFING: Certified teachers, volunteers 


DURATION: 10 weeks, 2 days/week, 1 hour/day 


IMPLEMENTATION SUPPORT: None reported 
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CAREER & LEADERSHIP PROGRAMS | STAY-IN-SCHOOL FOR COLLEGE AND CAREER OPPORTUNITIES (SISCO) 


Study Details 


STUDY DESCRIPTION 

The quasi-experiment estimates the impact of the Stay-in-School for College and Career Opportunities (SISCO). Outcomes for Latino 
middle school students (mostly in grades 7-8) who participated in SISCO for two years (2011-12 and 2012-13) are compared to outcomes 
for a matched comparison group of Latino middle school students who did not participate in the program. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; rage Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school : 7 
Tier Ill Tier Ill Grades 6-8 112 0.50 (0.19)* 


days attended 


Average effect size for Attendance & Enrollment outcomes at Tiers I-III | os 


* Statistically significant at p<.05 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome : F ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Promotion to ' 7 * 
Tier Ill Tier Ill Grades 6-8 112 0.50 (0.19) 


the next grade 


Average effect size for Promotion & Graduation outcomes at Tiers I-lll | os 


* Statistically significant at p<.05 


CITATION: Perry, J. C., & Shannon, L. (2017). How vocational psychologists can make a difference in K-12 education. International Journal of Educational 
and Vocational Guidance, 17, 97-115. 
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CAREER & LEADERSHIP PROGRAMS 


After School Matters 


After School Matters, offered in Chicago Public Schools, 
aimed to increase low-income urban adolescents’ knowledge 
of workplace culture and improve their soft skills through 
paid apprenticeships. Apprenticeships varied but were 
focused in particular areas and guided by instructors who 
provided information and feedback to students regarding 
workplace standards and culture. Instructors were not certified 
teachers but had expertise in the field of the apprenticeship. 
The implementation year is not reported. 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


School Engagement 

No Effect 

The study found neither improved nor negative outcomes for 
Attendance & Enrollment 
General Achievement 


Promotion & Graduation 


NOT MEASURED FOR THIS INTERVENTION: Mathematics 
Achievement, Other Achievement, Physical Activity/Health, Promotion 
& Graduation, Reading/ELA Achievement, Science Achievement, and 
Social & Emotional Competencies 


Program Content 
MAIN COMPONENT: Apprenticeships 


BEST PRACTICES USED: Sequenced, active learning 


Location 


LOCALE REGION 


Urban Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


92% Low-Income 77% Black 
23% Latino 


Logistics 

SETTING: School and community sites 

DELIVERY FORMAT: Group format 

STAFFING: Program staff 

DURATION: 10 weeks in fall and spring, 3 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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CAREER & LEADERSHIP PROGRAMS | AFTER SCHOOL MATTERS 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment, originally designed as a randomized controlled trial, estimates the impact of After School Matters for students in 


grades 9-12 in Chicago Public Schools. Outcomes for participants were compared to those for non-participants, many of whom partici- 
pated in afterschool activities or held jobs after school. The implementation year is not reported. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ’ or Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percentage of school 
2 co co Grades 9-12 193 0.09 (0.15) 


days attended 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 
co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; rae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
GPA 8 8 Grades 9-12 462 -0.05 (0.09) 


Average effect size for General Achievement outcomes at Tiers I-III 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome . , ror Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of courses 
8 8 Grades 9-12 462 -0.01 (0.09) 


failed 


Average effect size for Promotion & Graduation outcomes at Tiers I-lll 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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CAREER & LEADERSHIP PROGRAMS | AFTER SCHOOL MATTERS 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome , ; Te Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Identification with : . 
Tier Ill Tier Ill Grades 9-12 365 0.23 (0.11) * 


School Questionnaire 


Average effect size for School Engagement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


CITATION: Hirsch, B. J., Hedges, L. V., Stawicki, J. & Mekinda, M. A. (2011). Afterschool programs for high school students: An evaluation of After School 
Matters. Unpublished manuscript. 
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CAREER & LEADERSHIP PROGRAMS 


Hispanic Youth Leadership Program 


The Hispanic Youth Leadership Program, offered in one school 
district in 2011-13, aimed to provide low-income Latino high 
school students with leadership skills and social and emotional 
learning competencies to promote positive decision-making. 
The program engaged youth in service learning and drew ona 
curriculum developed by ASPIRA, a national Latino education 
organization, to teach leadership, career exploration, cultural 
enrichment, and social and emotional learning competencies. 
Homework help and tutoring were also offered. 


EFFECTIVENESS AT A GLANCE 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Attendance & Enrollment 


Promotion & Graduation 


No Effect 


The study found neither improved nor negative outcomes for 


General Achievement 


NOT MEASURED FOR THIS INTERVENTION: Mathematics 
Achievement, Other Achievement, Physical Activity/Health, Reading/ 
ELA Achievement, School Engagement, Science Achievement, and 
Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content 
MAIN COMPONENT: Tutoring 


BEST PRACTICES USED: Not reported 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Location 


LOCALE REGION 


Urban Midwest 


Participants 
CLASS GRADE 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Ue 


100% Latino 


TARGET STUDENTS 


Not Reported 


Logistics 
SETTING: School site 


DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers, volunteers 


DURATION: 10 weeks, 3 days/week, 2-2.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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CAREER & LEADERSHIP PROGRAMS | HISPANIC YOUTH LEADERSHIP PROGRAM 


Study Details 


STUDY DESCRIPTION 


This quasi-experiment estimates the impact of the Hispanic Youth Leadership Program (HYLP). Outcomes for high school students who 
participated in HYLP for two years, 2011-12 and 2012-13, are compared to outcomes for a matched comparison group of Latino high 
school students who did not participate in the program. The comparison students experienced “business as usual,” which could have 


included other school- and community-sponsored afterschool activities. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of 
Outcome . ; rae Group 
Without site and With site and students 
sample criteria sample criteria 
Number of school ‘ 7 
Tier Ill Tier Ill Grades 9-12 76 


days attended 


Average effect size for Attendance & Enrollment outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of 
Outcome ; ; rae Group 
Without site and With site and students 
sample criteria sample criteria 
GPA 8 8 Grades 9-12 76 


Average effect size for General Achievement outcomes at Tiers I-III 


Effect size 
(Standard error) 


0.62 (0.23)* 


Effect size 
(Standard error) 


0.32 (0.23) 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of 
Outcome . , ae Group 
Without site and With site and students 
sample criteria sample criteria 
High school graduation Tier Ill Tier Ill Grades 9-12 34 
Promotion to the next 
8 8 Grades 9-12 37 


grade 


Average effect size for Promotion & Graduation outcomes at Tiers I-lll 


* Statistically significant at p<.05 


Effect size 
(Standard error) 


1.43 (0.38)* 


0.26 (0.34) 


0.84 


§ Meets Tier Ill standards for cause-and-effect. Does not meet ESSA Tier Ill because there is not a statistically significant improvement in the outcome. 
CITATION: Perry, J. C., & Shannon, L. (2017). How vocational psychologists can make a difference in K-12 education. International Journal of Educational 


and Vocational Guidance, 17, 97-115. 
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CAREER & LEADERSHIP PROGRAMS 


LEAD Academy 


LEAD (Leadership Education and Development) Academy, 
offered at one school in an urban district in New Jersey in 
2005-06, aimed to improve conduct and promote leadership 
among academically at-risk fourth and fifth grade students. 
The program used a leadership curriculum (developed by the 
provider) that involved work stations focused on leadership 
skills. Small groups of students rotated through the work 
stations over a period of three weeks. Teachers, community 
leaders, entrepreneurs, and other guest speakers staffed the 
program. The program was offered for three weeks, five days 
per week. 


Negative Effect 
The study found at least one negative outcome with no 


overriding contrary evidence for 


Social & Emotional Competencies 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 
Physical Activity/Health, Promotion & Graduation, Reading/ELA 
Achievement, Attendance & Enrollment, School Engagement, and 
Science Achievement 


Program Content 
MAIN COMPONENTS: Social skills activities 


BEST PRACTICES USED: Active learning 


Location 


LOCALE REGION 


Urban Northeast 
Participants 
CLASS GRADE TARGET STUDENTS 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


71% Low-Income 100% Black 


Logistics 

SETTING: School site 

DELIVERY FORMAT: Group format 

STAFFING: Certified teachers, specialized staffing, program staff 
DURATION: 3 weeks, 5 days/week 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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CAREER & LEADERSHIP PROGRAMS | LEAD ACADEMY 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the Leadership Education and Development Academy (LEAD Academy) for fifth graders 
identified as being academically at risk. The program was offered at one school in an urban district in New Jersey, and students at a 
second school served as the comparison group. The comparison students experienced “business as usual,” which could have included 
other school- and community-sponsored afterschool activities. The study was conducted in 2005-06. 


SOCIAL & EMOTIONAL COMPETENCIES 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Leadership Skills 


Inventory: Decision . 
Making (Karnes & 8 8 Grade 5 34 -1.08 (0.37) 


Chauvin, 2000) 


Leadership Skills 
Inventory: Problem 


= * 

Solving (Karnes & oo oo Grade 5 34 1.24 (0.38) 
Chauvin, 2000) 
Leadership Skills 
Inventory: Written 

eee § § Grade 5 34 -1.20 (0.37)* 

Communication (Karnes 
& Chauvin, 2000) 
Average effect size for Social & Emotional Competencies outcomes at Tiers I-III -1.17 


* Statistically significant at p<.05 
co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


§ Meets Tier IIl standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Burton, M. D. (2006). The impact of a leadership development program on fifth-grade students’ disciplinary referrals and leadership 
(Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3206885) 
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CAREER & LEADERSHIP PROGRAMS 


Time Travelers Leadership Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Time Travelers Leadership Program, offered at one Location 
school in a suburban district in 2009-10, aimed to increase LOCALE REGION 
the school engagement, academic performance, and behavior 
of a diverse group of at-risk seventh grade students. The key 
components of the program focused on college and career 


preparation, leadership, and service. Extracurricular activities 


and homework help were also offered. The program was led 
by school teachers who received professional development on 


Suburban Midwest 
increasing school engagement. The program was offered for 
the full school year, two days per week, 2.5 hours each day. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 
General Achievement 
NOT MEASURED FOR THIS INTERVENTION: Mathematics 


Achievement, Other Achievement, Physical Activity/Health, 
Promotion & Graduation, Reading/ELA Achievement, Attendance & 
Enrollment, School Engagement, Science Achievement, and Social 75% White 
& Emotional Competencies 


69% Low-Income 25% Black 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Service, extracurriculars, SETTING: School site 


h k hel 
ee EN DELIVERY FORMAT: Group and one-to-one formats 


BEST PRACTICES USED: Not reported STAFFING: Certified teachers 
DURATION: Full school year, 2 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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CAREER & LEADERSHIP PROGRAMS | TIME TRAVELERS LEADERSHIP PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experimental study estimates the impact of the Time Travelers Leadership Program (TTLP) on student GPA. Outcomes for 
participants are compared to outcomes for a matched group of students who did not participate. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study was conducted 
in one school in a suburban district in the Midwest during 2009-10. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ‘ ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
GPA co co Grade 7 26 0.01 (0.40) 


Average effect size for General Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Fink, B. L. (2010). The effect of a seventh grade after school leadership program on the developmental assets, academic achievement, and 
behavior of non-thriving students (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3450199) 
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CAREER & LEADERSHIP PROGRAMS 


Ellas/Ellos Mentoring Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Ellas/Ellos program, offered in one school district in Location 
2011-13, was a career-focused mentoring program for Latino LOCALE REGION 
middle and high school students. Mentoring took place in 

individual as well as group formats. Group mentoring was 

gender-specific and involved guest speakers and participation 

in leadership and community service projects. Mentors were | | ij 


assigned to youth based on their career and other interests. 


Urban Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 
_ Attendance & Enrollment 
‘> General Achievement 

Promotion & Graduation 


NOT MEASURED FOR THIS INTERVENTION: Mathematics 


; : ; cat ; Not Reported 100% Latino 
Achievement, Other Achievement, Physical Activity/Health, Reading/ 
ELA Achievement, School Engagement, Science Achievement, and 
Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Tutoring, mentoring SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Certified teachers, volunteers 


DURATION: 4 individual sessions/month for 1 hour, 1 group 
session/month for 2.5 hours 


IMPLEMENTATION SUPPORT: None reported 
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CAREER & LEADERSHIP PROGRAMS | ELLAS/ELLOS MENTORING PROGRAM 


Study Details 


STUDY DESCRIPTION 

The quasi-experiment estimates the impact of a mentoring program for Latino high school students. Outcomes for Latino high school 
students who participated in mentoring for two years (2011-12 and 2012-13) are compared to a matched comparison group of students 
who did not participate. The comparison students experienced “business as usual,” which could have included other school- and 


community-sponsored afterschool activities. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ’ or Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school days 
- g g Grades 9-12 38 0.46 (0.33) 


attended 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; rae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
GPA 8 8 Grades 9-12 38 -0.06 (0.32) 


Average effect size for General Achievement outcomes at Tiers I-III 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome . , ror Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
High school graduation 8 § Grades 9-12 21 0.66 (0.45) 
Promotion to the next 
8 8 Grades 9-12 18 0.48 (0.48) 
grade 
Average effect size for Promotion & Graduation outcomes at Tiers I-lll 0.57 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Perry, J. C., & Shannon, L. (2017). How vocational psychologists can make a difference in K-12 education. International Journal of Educational 
and Vocational Guidance, 17, 97-115. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers (Fresno) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Fresno, California County Office of Education’s afterschool Location 

programs, offered from 2011-13, aimed to improve the LOCALE REGION 
academic performance of low-income students in grades 3-6. 
Programs included multiple types of academic assistance: 
tutoring, homework help, and instruction designed to reinforce 


school day activities. Bilingual academic support was also 


available. Enrichment activities, which rotated every six weeks, 
were aligned to student interests. The program was offered for 


Suburban, Rural, Urban West 
the full school year. 
Participants 
=a od ol Oe BAYA oe We cd hee CLASS GRADE TARGET STUDENTS 
Grades 3-6 Low-Income 


Positive Effect 


The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 


Mathematics Achievement 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, Attendance & Enrollment, School Engagement, Science 7% Black 6% White 
Achievement, and Social & Emotional Competencies 


45% Low-Income 77% Latino 9% Asian 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Tutoring, homework help, SETTING: School site 


ee 
rae peeve DELIVERY FORMAT: Group and one-to-one formats 


BEST PRACTICES USED: Not reported STAFFING: Certified teachers 
DURATION: Full school year, no other info provided 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (FRESNO) 


Study Details 


STUDY DESCRIPTION 
This quasi-experimental study estimates the impact of 21st Century Community Learning Center afterschool programs in Fresno, 
California, offered during 2011-13. The study compared outcomes for students who participated frequently in the afterschool programs 


to outcomes for non-participants. The comparison students experienced “business as usual,” which could have included other school- 
and community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; are Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards 7 _ 
- Tier Ill Tier Ill 2013 2,084 0.18 (0.04)* 
Test - mathematics 
California Standards ‘ A 
; Tier Ill Tier Ill 2012 1,130 0.22 (0.06)* 
Test - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-lll | oz 


READING/ELA ACHIEVEMENT 


* Statistically significant at p<.05 


ESSA evidence tier 
Number of Effect size 
Outcome i ; ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


California English 


Language 8 8 2012 402 0.11 (0.10) 
Development Test 


California Standards 


Test - English 8 8 2012 1,130 0.08 (0.06) 
language arts 


California Standards 


Test - English 8 8 2013 2,084 0.02 (0.04) 
language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.07 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Price, S. A. (2014). Academic outcomes of elementary afterschool programs: A matched-pair study within high-poverty, predominantly Latino 
student populations (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3702868) 
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MULTICOMPONENT PROGRAMS 


PARE Go iigaGunviiiniigg Mer lentitm aeolian 


(Philadelphia - Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


These 21st Century Community Learning Center programs, Location 


offered in Philadelphia in 2012-13, provided academic support LOCALE REGION 
and other enrichment activities such as athletics, art and music, 
recreation, and career and technical education. The aim was 

to improve academic and behavioral outcomes of low-income 
youth attending high-poverty, low-performing schools. The 
programs were offered for the full school year, five days per 


week, for up to three hours per day. 


Northeast 
EFFECTIVENESS AT A GLANCE 
Participants 
CLASS GRADE TARGET STUDENTS 
Positive Effect 
Grades 1-5 Low-Income 


The study found at least one improved outcome with no 
overriding contrary evidence for 
Attendance & Enrollment 


Mixed Effects 


This study found a mix of both improved and negative or null 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


outcomes for 
Mathematics Achievement 


Reading/ELA Achievement 
No Effect 


The study found neither improved nor negative outcomes for 
Science Achievement 84% Low-Income 52% Black 


33% Asian 


NOT MEASURED FOR THIS INTERVENTION: General Achievement, 
Other Achievement, Physical Activity/Health, Promotion & Graduation, 
School Engagement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Arts, tutoring, homework help, SETTING: School site 


academic enrichment, sports/recreation, mentoring DELIVERY FORMAT: Group format 


BEST PRACTICES USED: N 
aheeponed STAFFING: Program staff 
DURATION: Full school year, 5 days/week, 2.5-3 hours/day 
IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (PHILADELPHIA - ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool programs in Philadelphia offered by six 21st Century Community Learning 
Center grantees. Outcomes for participants who attended at least 30 days of the program are compared to outcomes for students who 
did not participate or who participated for fewer than 30 days. The comparison students experienced “business as usual,” which could 
have included other school- and community-sponsored afterschool activities. The study was conducted during 2012-13. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : A Sor. Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 

10 or more unexcused ‘ ? Cannot Cannot 
absences — 2012-13 mer erm Gredes'tS determine determine* 

10 or more unexcused Cannot Cannot 
absences — 2011-12 8 8 Grades +5 determine determine 


: : Cannot 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill : 
determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; eae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Grade of C or higher c ' Cc ' 
‘ ‘ anno anno 
lolmatie malice 8 8 Grades 1-5 determine determine 
—2012-13 
Mathematics course Cannot 
grade — 2011-12 8 8 Sides tS 1,051 determine 
Pennsylvania State Cc t c t 
anno anno 
Syst f A it — - A A 
olay aaah 8 8 Grades 1-5 determine determine 
mathematics — 2011-12 
Pennsylvania State 
Cannot 


System of Assessment - Tier Ill Tier Ill Grades 1-5 450 
mathematics — 2012-13 


determine* 


: 5 , A Cannot 
Average effect size for Mathematics Achievement outcomes at Tiers I-lll 


determine 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (PHILADELPHIA - ELEMENTARY) 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Out G Number of Effect size 
come ro 
; Without site and With site and me students (Standard error) 
sample criteria sample criteria 
Developmental Reading Cc ; C ; 
ee es : : cidest2 determine determine 
level — 2012-13 
Grade of C or higher in _ . Cannot Cannot 
reading — 2011-12 Ter Ten eiades ie determine determine* 
Pennsylvania State C 
System of Assessment 8 § Grades 1-5 453 Poise 
— reading — 2012-13 
Pennsylvania State Cc Cc 
gees Sea : : Gladest2 determin detennine 
— reading — 2011-12 
Reading course grade- Cannot 
2012-13 : : eiides t= Sd determine 


: : : : Cc t 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III ike 


determine 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Pennsylvania State 
System of Assessment § § Grades 1-5 265 Cannot 
F determine 
— science — 2012-13 
Science course Cannot 
r 1- 1,051 s 
grade — 2012-13 8 8 Glades 05 determine 


4 ‘ i , Cannot 
Average effect size for Science Achievement outcomes at Tiers I-lll 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Gao, J., Hallar, B., & Hartmann, T. A. (2014). A snapshot of OST programs in Philadelphia: An evaluation of eleven 21st Century Community 
Learning Center grantees. Philadelphia, PA: Research for Action. 


Hartmann, T. A., Gao, J., Kumar, A., & Edmunds, K. (2013). A snapshot of OST programming in Philadelphia: An evaluation of six 21st Century Community 
Learning Center grantees. Philadelphia, PA: Research for Action. 
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MULTICOMPONENT PROGRAMS 


After School Education and Safety (California — Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After School Education and Safety programs, offered in Location 
2006-11 and funded by the State of California, were designed 
to provide academic support and a safe environment after 


LOCALE REGION 


school for students in kindergarten through ninth grade. Sites 
provided both academic and non-academic enrichment, with 


specific activities varying across sites. Almost all sites reported 
offering physical fitness/sports and arts/music. The programs 
were offered for the full school year, with the number of hours 


West 
varying by program. 


EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 


Positive Effect Grades 3-5 Low-Income 


The study found at least one improved outcome with no 


ve ISG COA aly eMidene= toh SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Attendance & Enrollment 


Physical Activity/Health 


Negative Effect 


The study found at least one negative outcome with no 


overriding contrary evidence for 


Mathematics Achievement 


Reading/ELA Achievement 


79% Low-Income 67% Latino 8% Asian 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 13% White 2% Other 
ment, Other Achievement, Promotion & Graduation, School Engage- 11% Black 
ment, Science Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Homework help, arts/music, SETTING: School site 


nutrition/health, physical fitness/sports, social skills, tutoring DELIVERY FORMAT: Group and one-to-one formats 


BEST PRACTICES USED: N 
abhenomed STAFFING: Certified teachers, program staff, volunteers 
DURATION: Full school year, number of hours varies by program 


IMPLEMENTATION SUPPORT: Provider training 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL EDUCATION AND SAFETY (CALIFORNIA — ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact on academic achievement and attendance of the After School Education and Safety 


programs for students in California. Outcomes for participants were compared to those for non-participants. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. The study 


was conducted from 2006-11. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 


Number of Effect size 
Outcome : 7 are Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percentage of : ? 0.08 
Tier Ill Tier Ill r - 248,82 , 
days attended . eiades 35 8,820 (0.004)* 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 0.08 
* Statistically significant at p<.05 
MATHEMATICS ACHIEVEMENT 
ESSA evidence tier 
Number of Effect size 
Outcome : ' ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards -0.01 
; r = : 
Test - mathematics 8 8 Siades. 5 358,508 (0.003)* 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl | oo 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL EDUCATION AND SAFETY (CALIFORNIA — ELEMENTARY) 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome . ; rary Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Abdominal Strength Tier Il Tier Ill Grades 3-5 109,386 eee 
, (0.006)* 
Aerobic Capacity Tier Il Tier Ill Grades 3-5 109,386 O08 
: (0.006)* 
Body Composition Tier Il Tier Ill Grades 3-5 109,386 ~~ 
: (0.006)* 
Flexibility Tier Il Tier Il Grades 3-5 109,386 ee 
m (0.006)* 
Trunk Strength Tier I Tier Ill Grades 3-5 109,386 one 
: (0.006)* 
0.009 
Upper Body Strength 8 8 Grades 3-5 109,386 (0.006) 


Average effect size for Physical Activity/Health outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : Remy Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards 
-0.04 


Test — English cS) 8 Grades 3-5 358,699 


(0.003)* 
language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


§ Meets Tier IIl standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Huang, D., & Wang, J. (2012). Independent statewide evaluation of ASES and 21st CCLC afterschool programs: May 1, 2008-December 31, 
2011 (Report No. CDE4/CN077738/2011). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing, Center for the 
Study of Evaluation, Graduate School of Education and Information Studies, University of California, Los Angeles. 


Huang, D., Silver, D., Cheung, M., Duong, M., Gualpa, A., Hodson, C.,..., Vazquez, V. (2011). Independent statewide evaluation of after school programs 
(CRESST Report No. 789). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing, Graduate School of Education 
and Information Studies, University of California, Los Angeles. 
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MULTICOMPONENT PROGRAMS 


Baltimore Community Schools (Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Baltimore Community Schools programs, offered in 2012-14, Location 
took place at community schools serving low-income students LOCALE REGION 
in grades K-12. The programs were intended to promote 
academic success, health, social and emotional development, 
high school graduation, and college and career readiness. 
Programs were provided by various types of organizations 
and offered academic support, recreation, and arts and sports 


activities. The programs were offered for the full school year, 


Northeast 
up to five days per week, for up to 4.5 hours per day. 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades K-5 Low-Income 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Attendance & Enrollment 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical 
Activity/Health, Promotion & Graduation, Reading/ELA Achievement, 
School Engagement, Science Achievement, and Social & Emotional 9% Latino 
Competencies 


96% Low-Income 87% Black 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Sports, homework help SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Full school year, up to 5 days/week, 3-4.5 hours/day 


IMPLEMENTATION SUPPORT: Ongoing support 
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MULTICOMPONENT PROGRAMS | BALTIMORE COMMUNITY SCHOOLS (ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool programs in community schools administered by the Family League of 
Baltimore. Outcomes for students who attended afterschool programs for at least 200 hours were compared to outcomes for students 
who did not attend the programs and did not attend a community school. The comparison students experienced “business as usual,” 
which could have included other school- and community-sponsored afterschool activities. The study was conducted in 2012-14. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : A rears Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Chronic absenteeism Tier Il Tier Il ee era 2,584 0.53 (0.04)* 
K-5th graders 
Chronic absenteeism Tier Ill Tier Ill (new attendees)— 3,192 0.15 (0.04)* 
2012-13 
: ; K-5th graders 
Ch bsent 
oe ee Tier Il Tier Il (new attendees)— 1,702 0.35 (0.05)* 
(>20 days absent) 2013-14 
K-5th graders 
Chronic absenteeism Tier Ill Tier Ill (2-yr attendees)— 1,472 0.28 (0.05)* 
2013-14 
Percentage of school F . K-5th graders— Cannot 
days attended Tier Ill Tier Ill 2011-12 2,584 dcterminc® 
K-5th graders 
Percentage of school . ; Cannot 
days attended Tier Ill Tier Ill sa a 3,192 determine! 
K-5th graders 
P tt f school 
SACs Tier Il Tier I (new attendees)- 1,702 connee 
days attended 2013-14 determine 
K-5th graders 
Percentage of school ; : Cannot 
dayeatenced Tier Ill Tier Ill eae 1,472 devernne 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill | ose 


* Statistically significant at p<.05 


CITATIONS: Olson, L. S. (2014). A first look at community schools in Baltimore. Baltimore, MD: Baltimore Education Research Consortium. 


Olson, L. S., Connolly, F., & Kommajesula, A. H. (2013). Family League 2011-12 out of school time programs in Baltimore city. Baltimore, MD: Baltimore 
Education Research Consortium. 
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MULTI-STUDY PROGRAM SUMMARY 


LA’s Better Educated Students for Tomorrow (LA’s BEST) 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


Positive Effect 


LA’s BEST (Better Educated Students for Tomorrow) 
aimed to increase academic achievement and 
improve the interpersonal skills and self-esteem of 
elementary school children in high-crime, high- 
poverty areas in Los Angeles Unified School 
District. The programs included homework help 
and tutoring, other academically-oriented activities, 
sports and recreation, performing and visual arts, 
community and cultural activities, and health and 


nutrition programming. 


This review of LA’s BEST identified six studies of 
this program. The findings from each of these 


studies are presented individually on pages 174 — 182 


of this evidence guide. 


Chinen, M. H. (2010). The effects of afterschool program 
participation on mathematics achievement: The case of 
LA’s BEST (Doctoral dissertation, University of California, Los 
Angeles). Retrieved from ProQuest Dissertations and Theses 
Database (UMI No. 3450983) 


Goldschmidt, P., & Huang, D. (2007). The long-term effects 
of afterschool! programming on educational adjustment 

and juvenile crime: A study of the LA’s BEST afterschool 
program (CRESST Report). Los Angeles, CA: National Center 
for Research on Evaluation, Standards and Student Testing, 
Graduate School of Education and Information Studies, 


University of California, Los Angeles. 


Huang, D., Leon, S., Harven, A. M., La Torre, D., & Mostafavi, 
S. (2009). Exploring the relationship between LA’s BEST 
program attendance and cognitive gains of LA’s BEST 
students (CRESST Report No. 757). Los Angeles, CA: National 
Center for Research on Evaluation, Standards and Student 
Testing, Graduate School of Education and Information 


Studies, University of California, Los Angeles. 


Overall, the studies found at least one improved 
outcome with no overriding contrary evidence for 


Mathematics Achievement 


No Effect 


Overall, the studies found neither improved nor 
negative outcomes for 


Reading/ELA Achievement 
Other Achievement 


Science Achievement 


Huang, D., Kim, K. S., Marshall, A., & Pérez, P. (2005). 
Keeping kids in school: An LA’s BEST example. A study 
examining the long-term impact of LA’s BEST on students’ 
dropout rates (CRESST Report). Los Angeles, CA: National 
Center for Research on Evaluation, Standards and Student 
Testing, Graduate School of Education and Information 
Studies, University of California, Los Angeles. 


Huang, D., La Torre, D., Leon, S., Duong, N., & Hodson, 

C. (2011). Supporting student success in middle schools: 
Examining the relationship between elementary afterschool 
program participation and subsequent middle school 
attainments (CRESST Report). Los Angeles, CA: National 
Center for Research on Evaluation, Standards and Student 
Testing, Graduate School of Education and Information 
Studies, University of California, Los Angeles. 


Huang, D., Leon, S., La Torre, D., & Mostafavi, S. (2008). 
Examining the relationship between LA’s BEST program 
attendance and academic achievement of LA’s BEST stu- 
dents (CRESST Report No. 749). Los Angeles, CA: National 
Center for Research on Evaluation, Standards and Student 
Testing, Graduate School of Education and Information 


Studies, University of California, Los Angeles. 


MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #1 


LA’s Better Educated Students for Tomorrow (LA’s BEST — 1993-98) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


LA’s BEST (Better Educated Students for Tomorrow), offered Location 
at 24 Los Angeles schools in 1993-98, aimed to increase LOCALE REGION 


academic achievement and improve the interpersonal skills 
and self-esteem of elementary school children in high-crime, 
high-poverty areas in Los Angeles Unified School District. The 


programs included homework help and tutoring, other aca- 
demically-oriented activities, sports and recreation, performing 


and visual arts, community and cultural activities, and health West 


and nutrition programming. The programs were offered for the 
full school year, five days per week. 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 2-5 Low-Income 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, Attendance & Enrollment, School Engagement, Science 16% Black 1% Asian 
Achievement, and Social & Emotional Competencies 


71% Low-Income 81% Latino 2% White 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Tutoring, homework help, SETTING: School site 

mic enrichment. r ion n Itural 
academic enrichment, sports/recreation, arts and cultura DELIVERY FORMAT: Group format 
activities, health and nutrition education 


STAFFING: Program staff, volunteers 
BEST PRACTICES USED: Active learning, positive 


adult-student relationships, family engagement, respect for DURATION: Full school year, 5 days/week 


student cultures IMPLEMENTATION SUPPORT: Ongoing support 
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MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This study uses a quasi-experimental design to estimate the academic impacts of attending the LA’s BEST afterschool program during 
elementary school. The study examines reading and math achievement scores in 1998-99 (the start of middle school) among students 
who were in the second or third grade in the 1993-94 academic year. Outcomes for students who attended LA’s BEST program for at 
least one year during elementary school are compared to outcomes for students in the same grade and school who did not attend the 
LA’s BEST program during any elementary school year. The comparison students experienced “business as usual,” which could have 
included other school- and community-sponsored afterschool activities. The study was conducted in 24 schools. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . : ae: Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 
Stanford Achievement Cannot 

f r 2- 4,662 F 

Test 9 — mathematics 8 8 Grades 2-5 66 determine 


; : f ; Cannot 
Average effect size for Mathematics Achievement outcomes at Tiers I-lll 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome F . =e Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Stanford Achievement 


Test 9 — reading 8 8 Grades 2-5 4,662 alia 
comprehension 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III eannet 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Goldschmidt, P., & Huang, D. (2007). The long-term effects of afterschool programming on educational adjustment and juvenile crime: 
A study of the LA’s BEST afterschool program (CRESST Report). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student 
Testing, Graduate School of Education and Information Studies, University of California, Los Angeles. 


Huang, D., Kim, K. S., Marshall, A., & Pérez, P. (2005). Keeping kids in school: An LA’s BEST example. A study examining the long-term impact of LA’s BEST 
on students’ dropout rates (CRESST Report). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing, Graduate 
School of Education and Information Studies, University of California, Los Angeles. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 


175 


MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #2 


LA’s Better Educated Students for Tomorrow (LA’s BEST — 2001-04) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


LA’s BEST (Better Educated Students for Tomorrow), offered Location 
in 84 Los Angeles schools in 2001-04, aimed to increase 


LOCALE REGION 
academic achievement and improve the interpersonal skills 


and self-esteem of elementary school children in high-crime, 
high-poverty areas in Los Angeles Unified School District. The 


programs included homework help and tutoring, other aca- 
demically-oriented activities, sports and recreation, performing 


and visual arts, community and cultural activities, and health West 


and nutrition programming. The programs were offered for the 
full school year, five days per week, three hours per day. 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 3-4 Low-Income 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 


overriding contrary evidence for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Other Achievement, Physical Activity/Health, 


9 = 0, . 0, ; 
Promotion and Graduation, Reading/ELA Achievement, TN Loy eame ote Lanng amines 
Attendance & Enrollment, School Engagement, Science 5% Black 1% Other 
Achievement, and Social & Emotional Competencies 2% Asian 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help, SETTING: School site 


academic enrichment, sports/recreation, arts and cultural 
ee piace DELIVERY FORMAT: Group format 


activities, health and nutrition education 


STAFFING: Program staff, volunteers 
BEST PRACTICES USED: Positive adult-student relationships 


DURATION: Full school year, 5 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, ongoing support 
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MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experimental study estimates the impact of the LA’s BEST afterschool program for third and fourth grade students. Outcomes 
for program participants were compared to those who did not attend the program. The programs were offered for the full school year, 
five days per week. The study was conducted in 84 Los Angeles schools during 2001-04. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : aye Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards 
Test — mathematics — oo oo Grade 3 740 -0.01 (0.07) 
2001-02 
California Standards 
Test — mathematics — oo oo Grade 3 642 -0.01 (0.08) 
2002-03 
California Standards 
Test — mathematics — co oo Grade 3 340 -0.02 (0.11) 
2003-04 
California Standards 
Test — mathematics — co co Grade 4 256 -0.07 (0.13) 
2001-02 
California Standards 
Test — mathematics- oo oo Grade 4 358 0.00 (0.11) 


2002-03 


California Standards 
Test — mathematics- Tier Il Tier Ill Grade 4 294 0.23 (0.12) * 


2003-04 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill | 002 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


CITATION: Chinen, M. H. (2010). The effects of afterschool program participation on mathematics achievement: The case of LA’s BEST (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3450983) 
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MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #3 


LA’s Better Educated Students for Tomorrow (LA’s BEST — 2004-07) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


LA’s BEST (Better Educated Students for Tomorrow), offered in Location 


Los Angeles in 2004-07, aimed to increase academic achieve- LOCALE REGION 
ment and improve the interpersonal skills and self-esteem of 
elementary school children in high-crime, high-poverty areas 


in Los Angeles Unified School District. The programs included 


homework help and tutoring, other academically-oriented 
activities, sports and recreation, performing and visual arts, 


community and cultural activities, and health and nutrition 


West 

programming. The program was offered for the full school year, 
five days per week. 

Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 3-5 Low-Income 

SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effects 
The study found neither improved nor negative outcomes for 

Mathematics Achievement 
Reading/ELA Achievement 
NOT Menon ee FOR THIS NTE ENO: General INGIMENE? 40% of parents hiad less than 90% Latino 
ment, Other Achievement, Physical Activity/Health, Promotion & hivh echeoled ti 
Graduation, Attendance & Enrollment, School Engagement, Science pH Scnoohecucaton 5% Black 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help, SETTING: School site 
i ich tt i Itural 
academic enrichment, sports/recreation, arts and cultura DELIVERY FORMAT: Group format 
activities, health and nutrition education 
STAFFING: Program staff, volunteers 

BEST PRACTICES USED: Active learning, positive 
adult-student relationships, family engagement, respect for DURATION: Full school year, 5 days/week 
student cultures IMPLEMENTATION SUPPORT: Ongoing support 
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MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #3 


Study Details 


STUDY DESCRIPTION 
This study uses a quasi-experimental design to estimate the impacts of participating in LA’s BEST during third through fifth grade on 


math and reading achievement scores. The comparison group was students who participated in LA’s BEST during second grade but not 
in third through fifth grade. The study was conducted in 2004-07. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ape Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 
California Standards 

: co co Grades 2-5 5,793 0.04 (0.04) 

Test - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 
co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : : Tae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards 
: co co Grades 2-5 5,772 -0.04 (0.04) 
Test - reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATIONS: Huang, D., Leon, S., La Torre, D., & Mostafavi, S. (2008). Examining the relationship between LA’s BEST program attendance and academic 
achievement of LA’s BEST students (CRESST Report No. 749). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student 
Testing, Graduate School of Education and Information Studies, University of California, Los Angeles. 


Huang, D., Leon, S., Harven, A. M., La Torre, D., & Mostafavi, S. (2009). Exploring the relationship between LA’s BEST program attendance and cognitive 
gains of LA’s BEST students (CRESST Report No. 757). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing, 
Graduate School of Education and Information Studies, University of California, Los Angeles. 
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MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #4 


LA’s Better Educated Students for Tomorrow 
(LA's BEST — Long-Term Effects) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


LA’s BEST (Better Educated Students for Tomorrow) aimed Location 
to increase academic achievement and improve the interper- LOCALE REGION 


sonal skills and self-esteem of elementary school children in 
high-crime, high-poverty areas in Los Angeles Unified School 
District. The programs included homework help and tutoring, 


other academically-oriented activities, sports and recreation, 
performing and visual arts, community and cultural activities, 


and health and nutrition programming. West 
EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 


Positive Effect Grade 3 Low-Income 


The study found at least one improved outcome with no 
overriding contrary evidence for 


F , SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Mathematics Achievement 


No Effect 

The study found neither improved nor negative outcomes for 
Science Achievement 
Reading/ELA Achievement 
Other Achievement 


Not R % Lati 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- ot Reported 85% Latino 
ment, Physical Activity/Health, Promotion & Graduation, Attendance & 9% Black 
Enrollment, School Engagement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Tutoring, homework help, SETTING: School site 
academic enrichment, sports/recreation, arts and cultural 


DELIVERY FORMAT: Group format 
activities, health and nutrition education 


STAFFING: Program staff, volunteers 
BEST PRACTICES USED: Active learning, positive 


adult-student relationships, family engagement, respect for DURATION: Full school year, 5 days/week 


student cultures IMPLEMENTATION SUPPORT: Ongoing support 
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MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #4 


Study Details 


STUDY DESCRIPTION 

This study uses a quasi-experimental design to estimate the impacts of participating in LA’s BEST during third grade on course grades 
and achievement in eighth grade. The comparison group was students who did not participate in LA’s BEST during third grade. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 
activities. The study was conducted in 2004-07. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ee, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Grade 8 California Grade 3 Cc t 
7 i rade anno 
Standards Test — math- Tier Ill Tier Ill cohort 9,455 detavmina® 
ematics 
Grade 8 California § § Grade 3 9.455 Cannot 
Standards Test - algebra cohort , determine 
Grade 3 Cannot 
Grade 8 Algebra GPA cS} 8 cohort 12,756 determine 
Grade 8 General . . Grade 3 Cannot 
mathematics GPA aie ter cohort 12,756 determine* 


; F : Cannot 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


determine 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


OTHER ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i z waa Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Grade 8 history course r nn 
y g § Grade 3 12.756 Ca ot 
grade cohort determine 


: 3 : Cannot 
Average effect size for Other Achievement outcomes at Tiers I-Ill 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY LA’S BETTER EDUCATED STUDENTS FOR TOMORROW | SUMMARY #4 


READING/ELA ACHIEVEMENT 


Grade 8 California 


‘ Grade 3 Cannot 
Standards Test - English 8 § cohort 9,455 determine 
language arts 


Grade 8 English 
language arts § 8 
course grade 


Grade 3 


Cannot 
cohort 12,756 : 


determine 


Cannot 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III A 
determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCIENCE ACHIEVEMENT 


Grade 8 science Grade 3 


course grade 8 8 cohort 


Cannot 
12,756 determine 


Average effect size for Science Achievement outcomes at Tiers I-III Cannel 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier IIl because there is not a statistically significant improvement in the outcome. 


CITATION: Huang, D., La Torre, D., Leon, S., Duong, N., & Hodson, C. (2011). Supporting student success in middle schools: Examining the relationship 
between elementary afterschool program participation and subsequent middle school attainments (CRESST Report). Los Angeles, CA: National Center 
for Research on Evaluation, Standards and Student Testing, Graduate School of Education and Information Studies, University of California, Los Angeles. 
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MULTICOMPONENT PROGRAMS 


Multicomponent Program (Northeast) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This afterschool program, offered during 2002-04 in three Location 
public schools in economically disadvantaged communities LOCALE REGION 
in the Northeast, aimed to provide a safe environment and 
promote academics, social skills, and physical health to 
students in grades 1-3. Programming involved a mix of 
homework help, enrichment activities (e.g., computers, visitors, 


musical instruments), recreational activities (e.g., sports, board 


games), and art. The program was offered for the full school Noedieact 


year, five days per week, three hours each day. 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 1-3 Low-Income 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Physical Activity/Health 


Social & Emotional Competencies 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 95% Low-lacomeé 46% Latino 6% White 
ment, Mathematics Achievement, Other Achievement, Promotion & 

Graduation, Reading/ELA Achievement, Attendance & Enrollment, 34% Black 14% Other 
School Engagement, and Science Achievement 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Arts, sports, homework help SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers, program staff 
DURATION: Full school year, 5 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | MULTICOMPONENT PROGRAM (NORTHEAST) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impacts of an afterschool program for first through third graders at three schools in the Northeast. 
Outcomes for “regular participants” (those who attended the program three days per week on average) were compared to outcomes for 
those who did not attend, or “sporadic attenders.” The study was conducted in 2002-03 and 2003-04. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; j Pas) Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index Tier Il Tier Ill Grades 1-3 364 Cannot 
determine* 
Obesity (>=95th : : 
Ba Tier Ill Tier III Grades 1-3 364 0.22 (0.11)* 


percentile BMI) 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill | om 


SOCIAL & EMOTIONAL COMPETENCIES 


* Statistically significant at p<.05 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; aa Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Interpersonal 
Competence Scale Tier Ill Tier Ill Grades 1-3 362 Cannot 
determine 


(teacher rating) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III Saunt 


determine 


* Statistically significant at p<.05 


CITATION: Mahoney, J. L., Lord, H., & Carryl, E. (2005). Afterschool program participation and the development of child obesity and peer acceptance. 
Applied Developmental Science, 9, 202-215. 
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MULTICOMPONENT PROGRAMS 


Tutoring and Enrichment Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This tutoring and enrichment program, offered in one elemen- Location 
tary school in the Southeast in 2005-06, aimed to improve the LOCALE REGION 
academic performance, behavior, and attitude of low-achieving 
elementary school students. Participants received individual 
and small-group tutoring and participated in enrichment 
activities such as dance and music. Tutors were volunteers 
from a local university. The program was offered for 12 weeks, 
four r k, 1.75 hour: r é 
Sil aysiPEl Weel (> neue perday Not Reported Southeast 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 3-5 Low-Achieving Students 
Positive Effect 
The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 


Mathematics Achievement 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 

Not Reported 85% Black 
ment, Other Achievement, Physical Activity/Health, Promotion & P i 
Graduation, Attendance & Enrollment, School Engagement, Science 15% White 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Tutoring, enrichment activities SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Not reported 
DURATION: 12 weeks, 4 days/week, 1.75 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | TUTORING AND ENRICHMENT PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of this afterschool program in one elementary school in the Southeast, for students who 
scored below proficient on standardized tests. The comparison group is comprised of students who did not participate in the program. 
The comparison students experienced “business as usual,” which could have included other school- and community-sponsored after- 
school activities. The study was conducted in 2005-06. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ‘ rae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Mathematics : é 
Tier Ill Tier Ill Grades 3-5 20 1.10 (0.48)* 


course grade 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 
* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Reading course grade co oo Grades 3-5 20 0.94 (0.47) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Walker, C. S., Kronick, R., & Diambra, J. F. (2007). Assessment of a full-service school, after-hours tutoring and enrichment program. Journal 
of At-Risk Issues, 13(2), 21-27. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers 


(Marietta Boys and Girls Club) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Marietta Boys and Girls Club’s 21st Century Community Location 


Learning Center program, offered in one school, aimed to help LOCALE REGION 
low-income seventh and eighth grade students reach profi- 
ciency in mathematics or reading. The program, a collaboration 
between the Marietta, Georgia Public Schools and the Club, 
provided remedial academic support as well as enrichment 


to prepare for standardized tests. The program also offered 


personal growth and social development opportunities and 


Southeast 
encouraged family involvement in education activities. The 
program was offered for the full school year, five days per 
week, two hours each day. 
Participants 
CLASS GRADE TARGET STUDENTS 
Low-Income, 
EFFECTIVENESS AT A GLANCE Grades 7-8 leit 
Mixed Effects SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


The study found a mix of improved outcomes and null or 


negative outcomes for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


: : a ; 80% Low-Income 40% Black 2% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & . 
Graduation, Attendance & Enrollment, School Engagement, Science 40% Latino 2% Other 
Achievement, and Social & Emotional Competencies 17% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Tutoring, enrichment activities SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Full school year, 3 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (MARIETTA BOYS AND GIRLS CLUB) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool program in one school in Georgia in 2008-09. Outcomes for participants 
were compared to those for non-participants. The comparison students experienced “business as usual,” which could have included 
other school- and community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; j Pas) Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced 
cS} cS} Grade 7 82 -1.03 (0.24)* 
Competency Test - 
mathematics 
Georgia Criterion- 
Referenced : : 
Tier Il Tier Ill Grade 8 58 0.64 (0.27)* 


Competency Test - 
mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : y ste Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced : , 
Tier Il Tier Ill Grade 7 58 0.85 (0.27)* 
Competency Test - 
English language arts 
Georgia Criterion- 
Referenced 
cS} 8 Grade 8 81 -119 (0.24)* 
Competency Test - 
English language arts 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-IIl -0.17 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Venzen, M. A. (2011). Impact of the 21st Century afterschool program on student achievement in mathematics and language arts (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3487238) 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers (Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This is a national study of the impact of 21st Century Commu- 


nity Learning Centers in 2000-01. Programs served low-income Location 
middle school students in school districts around the country, LOCALE REGION 
including those in rural and urban areas, and aimed to provide 
academic support, recreation, and enrichment activities. i 
Programs typically operated in school buildings but were 
sometimes run by community organizations. Programs offered B 
a range of options, including homework help, academic 
tiviti hleti n Itural activities, life skill ining, , 
activities, athletics, arts and cultural activities, life skills training Ruta, Substban, Uiban Allregions 


and unstructured recreational time. Programs were offered for 
the full school year, five days per week, three hours each day. 


EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 


Positive Effect 


F : - Low-I 
The study found at least one improved outcome with no Grades 6-8 Oi eee ta 


overriding contrary evidence for 


Attendance & Enrollment 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


School Engagement 
Negative Effect 


The study found at least one negative outcome with no 
overriding contrary evidence for 
Other Achievement 


No Effect 


The study found neither improved nor negative outcomes for 
Mathematics Achievement 


At most schools, >50% of 33% White 28% Latino 
students eligible for free or 28% Black 7% Other 
Reading/ELA Achievement reduced-price lunch 


Science Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


ment, Physical Activity/Health, Promotion & Graduation, and Social & 
Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic tutoring, homework SETTING: School site 


hel i ial | 
elp, arts, sports/recreation, social development DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Not d 
i STAFFING: Certified teachers, program staff 

DURATION: Full school year, 5 days/week, 3 hours/day 

IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool programs offered to students in grades 6-8 at 62 sites in 34 school districts. 
Outcomes for students who participated in an afterschool program are compared to outcomes for students who did not participate. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 
activities. The study was conducted in 2000-01. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome é ¥ ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school : : 
Tier Il Tier Il Grades 6-8 4189 0.11 (0.03) * 
days absent 
Number of school : : Cannot 
Hayelats Tier Il Tier Il Grades 6-8 4189 determine! 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : pone Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Mathematics 
co oo Grades 6-8 4,189 0.06 (0.03) 


course grade 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 
co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome s ; en Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
English course grade co oo Grades 6-8 4,189 0.01 (0.03) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (MIDDLE GRADES) 


OTHER ACHIEVEMENT 


Social studies 
course grade 


oo oo Grades 6-8 4189 0.03 (0.03) 


Average effect size for Other Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


SCHOOL ENGAGEMENT 


Completes assignments 
to teacher’s satisfaction Tier Il Tier Il Grades 6-8 4,89 0.12 (0.03) * 
(teacher report) 


Student effort 


composite (teacher report) ver Tier Il Grades 6-8 4,189 0.10 (0.03) 


Usually tries hard in 
reading or English Tier Il Tier Il Grades 6-8 4,89 0.07 (0.03) * 
(teacher report) 


Average effect size for School Engagement outcomes at Tiers I-III 


* Statistically significant at p<.05 


SCIENCE ACHIEVEMENT 


Science course grade co oo Grades 6-8 4,89 0.01 (0.03) 


Average effect size for Science Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATIONS: Dynarski, M., Moore, M., Mullens, J., Gleason, P., James-Burdumy, S., Rosenberg, L., .. ., Deke, J. (2003). When schools stay open late: The 
national evaluation of the 21st Century Community Learning Centers program. First year findings (Report No. ED473230). Princeton, NJ: Mathematica 
Policy Research. 


Dynarski, M., Moore, M., James-Burdumy, S., Rosenberg, L., Deke, J., & Mansfield, W. (2004). When schools stay open late: The national evaluation of the 
21st Century Community Learning Centers program. New findings (MPR Report No. 8658-800/8720-700). Princeton, NJ: Mathematica Policy Research. 


James-Burdumy, S., Dynarski, M., & Deke, J. (2007). When elementary schools stay open late: Results from the national evaluation of the 21st Century 
Community Learning Centers program. Educational Evaluation and Policy Analysis, 29(4), 296-318. 


James-Burdumy, S., Dynarski, M., & Deke, J. (2008). Afterschool program effects on behavior: Results from the 21st Century Community Learning Centers 
program national evaluation. Economic Inquiry, 46(1), 13-18. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers 


(Philadelphia - Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


These 21st Century Community Learning Centers programs, Location 


offered in Philadelphia in 2012-13, provided academic support LOCALE REGION 
and other enrichment activities such as athletics, art and music, 
recreation, and career and technical education. The aim was 

to improve academic and behavioral outcomes of low-income 
youth attending high-poverty, low-performing schools. The 
programs were offered for the full school year, five days per 
week, for up to three hours per day. 


Northeast 

EFFECTIVENESS AT A GLANCE 

Participants 

CLASS GRADE TARGET STUDENTS 
Positive Effect 
The study found at least one improved outcome with no Grades 6-8 Lue 
overriding contrary evidence for 

Mathematics Achievement SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mixed Effects 


This study found a mix of both improved and negative or null 
outcomes for 


Reading/ELA Achievement 
No Effect 


The study found neither improved nor negative outcomes for 


Attendance & Enrollment 83% Low-Income 72% Black 


22% Latino 
Science Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Other Achievement, Physical Activity/Health, 
Promotion & Graduation, School Engagement, and Social & Emotional 
Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Arts, tutoring, homework help, SETTING: School site 


academic enrichment, sports/recreation, mentoring DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Not reported STAFFING: Program staff 
DURATION: Full school year, 5 days/week, 2.5-3 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (PHILADELPHIA - MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool programs in Philadelphia offered by six 21st Century Community Learning 
Center grantees. Outcomes for participants who attended at least 30 days of the program are compared to outcomes for students who 
did not participate or who participated for fewer than 30 days. The comparison students experienced “business as usual,” which could 
have included other school- and community-sponsored afterschool activities. The study was conducted during 2012-13. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ey Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
10 or more unexcused nn nn 
cS} 8 Grades 6-8 Cannot Cannot 
absences — 2011-12 determine determine 
10 or more unexcused nn nn 
8 8 Grades 6-8 Cannot Cannot 
absences — 2012-13 determine determine 


; : Cannot 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill i 
determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : : poe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Grade of C or higher Cc ' c t 
F , anno anno 
in mathematics 8 8 Grades 6-8 detcrmnine determine 
2011-12 
Mathematics Cc t 
a ‘ anno 
course grade — Tier Ill Tier Ill Grades 6-8 1/24 deterniine! 
2012-13 
Pennsylvania State 
System of Assessment : ' Cannot Cannot 
— mathematics — oo) em Grades 6-8 determine determine* 
2011-12 
Pennsylvania State 
System of Assessment Cannot 
r - 1157 : 
— mathematics- 8 8 sindese2 5 determine 


2012-13 


? : ; : Cannot 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


determine 


* Statistically significant at p<.05 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (PHILADELPHIA - MIDDLE GRADES) 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome , ; Pare Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Grade of C or higher in Cannot Cannot 
reading — 2011-12 8 8 Grades 6's determine determine 
Pennsylvania 
State System of Cannot Cannot 
Assessment — 8 8 Grades 6-8 determine determine 
reading — 2011-12 
Pennsylvania 
State System of Cannot 
Assessment — S 8 Siaaee aie 1155 determine 
reading — 2012-13 
Bese cui aiedes Tier I Tier Ill Grades 6-8 1124 Sannot 
2012-13 ba - eae determine* 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III Cannot 


determine 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; A a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Pennsylvania State 
System of Assessment § § Grades 6-8 231 Cannot 
: determine 
— science — 2012-13 
Science course grade — Cannot 
2012-13 8 8 Grades GS ble determine 


: : F : Cannot 
Average effect size for Science Achievement outcomes at Tiers I-Ill 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Gao, J., Hallar, B., & Hartmann, T. A. (2014). A snapshot of OST programs in Philadelphia: An evaluation of eleven 21st Century Community 
Learning Center grantees. Philadelphia, PA: Research for Action. 


Hartmann, T. A., Gao, J., Kumar, A., & Edmunds, K. (2013). A snapshot of OST programming in Philadelphia: An evaluation of six 21st Century Community 
Learning Center grantees. Philadelphia, PA: Research for Action. 
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MULTICOMPONENT PROGRAMS 


After School Education and Safety 


(California — Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After School Education and Safety programs, offered in Location 


2006-11 and funded by the State of California, were designed LOCALE REGION 
to provide academic support and a safe after school environ- 
ment for students in kindergarten through ninth grade. Sites 
provided both academic and non-academic enrichment, with 


specific activities varying across sites. Almost all sites reported 
offering physical fitness/sports and arts/music. The programs 
were offered for the full school year, with the number of hours 


West 
varying by program. 


EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 


Positive Effect Grades 6-8 Low-Income 


The study found at least one improved outcome with no 


overriding contrary evidence for 
SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Attendance & Enrollment 
Mathematics Achievement 


Physical Activity/Health 


No Effect 
The study found neither improved nor negative outcomes for 

Reading/ELA Achievement 79% Low-Income 67% Latino 8% Asian 
NOT MEASURED FOR THIS INTERVENTION: General 13% White 2% Other 
Achievement, Other Achievement, Physical Activity/Health, 11% Black 


Promotion & Graduation, School Engagement, and Social & Emotional 
Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Homework help, arts/ SETTING: School site 
music, nutrition/health, physical fitness/sports, 


DELIVERY FORMAT: Group and one-to-one formats 
social skills, tutoring 


STAFFING: Certified teachers, program staff, volunteers 
BEST PRACTICES USED: Not reported 
DURATION: Full school year, dosage varies by activity 


IMPLEMENTATION SUPPORT: Provider training 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL EDUCATION AND SAFETY (CALIFORNIA — MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact on academic achievement and attendance of the After School Education and Safety 


Program for students in California. Outcomes for participants were compared to those for non-participants. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. The study 


was conducted from 2006-11. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percentage of : ‘ x 
Tier Ill Tier Ill Grades 6-8 216,196 0.11 (0.004) 


days attended 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill | on 


MATHEMATICS ACHIEVEMENT 


* Statistically significant at p<.05 


ESSA evidence tier 
Number of Effect size 
Outcome ; . aerew Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards a - 
Tier Ill Tier Ill Grades 6-8 313,270 0.01 (0.004)* 


Test - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl | oo 


* Statistically significant at p<.05 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL EDUCATION AND SAFETY (CALIFORNIA — MIDDLE GRADES) 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome . ; rary Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
: , ; 0.05 
Abdominal Strength Tier Ill Tier Ill Grades 6-8 136,640 (0.005)* 
; : : , 0.09 
Aerobic Capacity Tier Ill Tier Ill Grades 6-8 136,640 (0.005)* 
Re : : 0.02 
Body Composition Tier Ill Tier Ill Grades 6-8 136,640 (0.005)* 
Heer : : 0.02 
Flexibility Tier Ill Tier Ill Grades 6-8 136,640 (0.005)* 
: : 0.04 
Trunk Strength Tier Ill Tier Ill Grades 6-8 136,640 (0.005)* 
: : 0.04 
Upper Body Strength Tier Ill Tier Ill Grades 6-8 136,640 (0.005)* 


Average effect size for Physical Activity/Health outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Standards 
-0.002 


Test a English § § Grades 6-8 314128 (0.004) 
language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Huang, D., & Wang, J. (2012). Independent statewide evaluation of ASES and 21st CCLC afterschool programs: May 1, 2008-December 31, 
2011 (Report No. CDE4/CN077738/2011). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing, Center for the 
Study of Evaluation, Graduate School of Education and Information Studies, University of California, Los Angeles. 


Huang, D., Silver, D., Cheung, M., Duong, M., Gualpa, A., Hodson, C., ..., Vazquez, V. (2011). Independent statewide evaluation of after school programs 
(CRESST Report No. 789). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing, Graduate School of Education 
and Information Studies, University of California, Los Angeles. 
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MULTICOMPONENT PROGRAMS 


The After-School Corporation (Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After-School Corporation programs, offered in New York Location 
City in 1998-2002, aimed to increase academic learning LOCALE REGION 
and healthy development and reduce anti-social behavior. 
Programs were school-based but supported by community 
organizations. Activities varied but included academic support, 
arts and culture, physical activities, health education, commu- 


nity service, and technology. Older youth also received college 


and career preparation, internships, peer counseling, and 


Northeast 
violence prevention. Programs were offered for the full school 
year, five days per week, three hours each day. 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 6-8 Low-Income 


Positive Effect 


The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
overriding contrary evidence for 


Attendance & Enrollment 


Mixed Effects 


The study found a mix of improved outcomes and null or 
negative outcomes for 
Promotion & Graduation 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- ; 87% Low=-lncoimeé 45% Latino 8% Asian 
ment, Mathematics Achievement, Other Achievement, Physical Activity/ 

Health, Reading/ELA Achievement, Engagement with Schooling, 41% White 6% White 
Science Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic enrichment, arts, health SETTING: School site 


education, sports/recreation, other enrichment 
ueation; Sperm /recrestion, ether Shnchime DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Sequenced, active learning, 


positive adult-student relationships, family engagement, 
respect for student cultures DURATION: Full school year, 5 days/week, 3 hours/day 


STAFFING: Certified teachers, program staff 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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MULTICOMPONENT PROGRAMS | THE AFTER-SCHOOL CORPORATION (MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates program impacts for students in grades 6-8 who participated in one of New York City’s 28 After- 
School Corporation programs between 1988-99 and 2001-02. The analysis includes only students who participated for at least one 
year, attended the program for at least 60 days and at least 60 percent of those days, and remained in the New York City public school 
system through ninth grade. The comparison group includes students from the same schools who did not participate in the program. 
The comparison students experienced “business as usual,” which could have included other school- and community-sponsored after- 


school activities. 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome ; ¥ apa Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Credits earned - 
Oth atace § § Grades 6-8 1,375 0.07 (0.05) 
Credits earned - 
ieeene § § Grades 6-8 526 -0.20 (0.09) 
11th grade 
Credit d- fs a 
pices Tier Ill Tier Ill Grades 6-8 2,582 0.12 (0.04) * 
9th grade 
On-time promotion from Cannot 
9th to 10th grade 8 8 Glades <2 determine 0.05 


Average effect size for Promotion & Graduation Outcomes at Tiers I-lll | cor 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 199 


MULTICOMPONENT PROGRAMS | THE AFTER-SCHOOL CORPORATION (MIDDLE GRADES) 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome A ; rary Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Continued enrollment Cc i 
anno 
two years after 8 8 Grades 6-8 detec 0.08 
9th grade 
Percentage of school 
days attended 8 cS} Grades 6-8 417 0.04 (0.10) 
12th grade 
Percentage of school 
days attended Tier Ill Tier Ill Grades 6-8 2,459 0.18 (0.04) * 
10th grade 
Percentage of school 
days attended Tier Ill Tier Ill Grades 6-8 3,821 0.26 (0.03) * 


9th grade 


Percentage of school 
days attended 8 8 Grades 6-8 1,08 0.08 (0.06) 


11th grade 
Average effect size for Attendance & Enrollment outcomes at Tiers I-lll pos 


§ Meets Tier IIl standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


CITATION: Russell, C. A., Mielke, M. B., Miller, T. D., & Johnson, J. C. (2007). Afterschool programs and high school success: Analysis of post-program 
educational patterns of former middle-grades TASC participants. Washington, DC: Policy Studies Associates, Inc. 
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MULTICOMPONENT PROGRAMS 


Nia Asils 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


AfterZone, offered from 2008-10, was a neighborhood-based 
network of afterschool programs in Providence, Rhode Island Location 
that aimed to provide a coordinated set of afterschool options 

LOCALE REGION 
for middle school students. Some activities took place on 
school campuses, and other activities took place in community 
settings. Students who signed up for community-activities 
received transportation to the activity. Participants chose from 


a variety of arts, skills, and sports programs and participated 


in academic enrichment. Programs were offered for the full 


Northeast 
school year, four days per week, 2.5 hours each day. 
EFFECTIVENESS AT A GLANCE 
Participants 
Positive Effect CLASS GRADE TARGET STUDENTS 
The study found at least one improved outcome with no 
overriding contrary evidence for Grades 6-8 Low-Income 
Attendance & Enrollment 
Mixed Effects SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


This study found a mix of both improved and negative or null 
outcomes for 
Physical Activity/Health 


No Effect 


The study found neither improved nor negative outcomes for 
Mathematics Achievement 


Reading/ELA Achievement 


92% Low-Income 56% Latino 4% White 
Scticol Engagement 13% Black 21% Other 
Science Achievement 5% Asian 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Promotion & Graduation, and Social & 
Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic enrichment, arts, SETTING: School and community sites 
sports, social development 


DELIVERY FORMAT: Group format 
BEST PRACTICES USED: Active learning, positive 


STAFFING: Certified teachers, program staff 
adult-student relationships 


DURATION: Full school year, 4 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: Provider training, ongoing support 
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MULTICOMPONENT PROGRAMS | AFTERZONE 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact of the AfterZone program in Providence, Rhode Island in 2008-09 and 2009-10. The 


comparison group is comprised of students who did not participate in the AfterZone program. About a third of these comparison 
students participated in other afterschool programs, many of which were academically-oriented. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; p rare Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percent of school 5 : 
Tier Il Tier Il Grades 6-8 763 0.15 (0.07)* 
days absent 
Percent of school : . 
Tier Il Tier Il Grades 6-8 763 0.15 (0.07) * 


days tardy 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill pow 


MATHEMATICS ACHIEVEMENT 


* Statistically significant at p<.05 


ESSA evidence tier 


Number of Effect size 
Outcome ; r aoe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Mathematics 
co co Grades 6-8 763 0.03 (0.07) 


course grade 


New England Common 
Assessments Program oo co Grades 6-8 763 -0.01 (0.07) 


— mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl | cor 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 202 


MULTICOMPONENT PROGRAMS | AFTERZONE 


PHYSICAL ACTIVITY/HEALTH 


D k 

By eee scen & co Grades 6-8 763 0.10 (0.07) 

ate fruit 
Days/week student ate o os Grades 6-8 763 -0.03 (0.07) 
sweet snacks 

Hours of physical i . : 
Bete ne ree Tier Il Tier Il Grades 6-8 763 0.15 (0.07) 
H f 

ours of sedentary co - Grades 6-8 763 -0.14 (0.07) 


activity in last week 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill | 002 | 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


English language arts 


oo oo Grades 6-8 763 0.04 (0.07) 
course grade 


New England Common 
Assessments Program eo oo Grades 6-8 763 -0.04 (0.07) 


— reading 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III | 000 


oo Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | AFTERZONE 


SCHOOL ENGAGEMENT 


Doing well in school is 
important to student co oo Grades 6-8 763 0.14 (0.07) 
(self-report) 


Hours on homework in 


oo oo Grades 6-8 763 0.06 (0.07) 
last week 


Average effect size for Schoo! Engagement outcomes at Tiers I-lll 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


SCIENCE ACHIEVEMENT 


Science course grade oo oo Grades 6-8 763 0.08 (0.07) 


Average effect size for Science Achievement outcomes at Tiers I-Ill 0.08 


oo Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Kauh, T. J., (2011). AfterZone: Outcomes for youth participating in Providence’s citywide afterschool system. 
Philadelphia, PA: Public/Private Ventures. 
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MULTICOMPONENT PROGRAMS 


Baltimore Community Schools (Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Baltimore Community Schools programs, offered in 2012-14, Location 
took place at community schools serving low-income students LOCALE REGION 
in grades K-12. The programs were intended to promote 
academic success, health, social and emotional development, 
high school graduation, and college and career readiness. 
Programs were provided by various types of organizations 
and offered academic support, recreation, and arts and sports 


activities. The programs were offered for the full school year, 


Northeast 
up to five days per week, for up to 4.5 hours per day. 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 6-8 Low-Income 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Attendance & Enrollment 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical Activity/ 
Health, Promotion & Graduation, Reading/ELA Achievement, Engage- 
ment with Schooling, Science Achievement, and Social & Emotional 5% Latino 
Competencies 


96% Low-Income 88% Black 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Sports, homework help, SETTING: School site 


‘ : 
arts, recreation DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Not reported STAFFING: Program staff 
DURATION: Full school year, up to 5 days/week, 3-4.5 hours/day 


IMPLEMENTATION SUPPORT: Ongoing support 
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MULTICOMPONENT PROGRAMS | BALTIMORE COMMUNITY SCHOOLS (MIDDLE GRADES) 


This quasi-experiment estimates the impact of afterschool programs in community schools, administered by the Family League of 
Baltimore. Outcomes for students who attended afterschool programs for at least 200 hours were compared to outcomes for students 
who did not attend the programs and did not attend a community school. The comparison students experienced “business as usual,” 
which could have included other school- and community-sponsored afterschool activities. The study was conducted in 2012-14. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; eat Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Chronic absenteeism 6th-8th grader 
(more than 20 8 8 (new attendees) — 180 -0.10 (0.15) 
days absent) 2013-14 
6th-8th graders 
Chronic absenteeism 8 8 (new attendees) — 426 0.17 (0.10) 
2012-13 
Chronic absenteeism Tier Ill Tier III 6th-8th graders — 624 0.50 (0.08)* 
2011-12 
6th-8th graders 
Chronic absenteeism 8 8 (2-yr attendees) — 254 0.16 (0.13) 
2013-14 
Percentage of school . , 6th-8th graders — Cannot 
Gay setended Tier Ill Tier Ill 2011-12 624 determing 
6th-8th graders 
Percentage of school 2 : Cannot 
Gaye sitenved Tier Ill Tier Ill (2-yr oa - 254 detarminc® 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill | os 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


CITATIONS: Olson, L. S. (2014). A first look at community schools in Baltimore. Baltimore, MD: Baltimore Education Research Consortium. 


Olson, L. S., Connolly, F., & Kommajesula, A. H. (2013). Family League 2011-12 out of school time programs in Baltimore city. Baltimore, MD: Baltimore 
Education Research Consortium. 
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MULTICOMPONENT PROGRAMS 


Cooke Middle School Afterschool Recreation Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This afterschool recreation program, offered at Cooke Middle Location 
School in Philadelphia, Pennsylvania in 2000-01, aimed to 

LOCALE REGION 
improve academic and social outcomes for low-income middle 
school youth ages 10-14. The program was intentionally 
unstructured and offered use of the gymnasium or the fitness 
center, arts and crafts, and dance. In addition, students could 
access an open classroom to play board games, read, or do 


homework. The program was offered for six months, four days 


Northeast 
per week, two hours each day. 
EFFECTIVENESS AT A GLANCE Participants 
CLASS GRADE TARGET STUDENTS 
Positive Effect Grades 5-8 Low-Income 


The study found at least one improved outcome with no 
overriding contrary evidence for 


: we SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Physical Activity/Health 


School Engagement 


No Effect 


The study found neither improved nor negative outcomes for 
Attendance & Enrollment 


General Achievement 
Not Reported 83% Black 2% White 
3% Asian 12% Other 


Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: Other Achievement, 


Promotion & Graduation, Science Achievement, and Social & Emotional 
Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Arts, recreation SETTING: School site 
BEST PRACTICES USED: Positive adult- DELIVERY FORMAT: Group format 


SHigenn relationships STAFFING: Certified teachers 


DURATION: 6 months, 4 days/week, 2 hours/day 
IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | COOKE MIDDLE SCHOOL AFTERSCHOOL RECREATION PROGRAM 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment, originally designed as a randomized controlled trial, estimates the impact of an afterschool program at one 


middle school in Philadelphia. Outcomes for participants were compared to those for non-participants. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. The study 
was conducted in 2000-01. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 


Number of Effect size 
Outcome . j weg: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percent of days 
: § § Grades 5-8 218 -0.06 (014) 


attended 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ~ Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
GPA — 3rd 
: : 8 8 Grades 5-8 207 -0.06 (0.14) 
marking period 
GPA — 4th 
8 8 Grades 5-8 216 0.04 (0.14) 


marking period 
Average effect size for General Achievement outcomes at Tiers I-IIl 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome E : Tamas Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Stanford Achievement 
: 8 8 Grades 7-8 128 -0.07 (0.18) 
Test 9 — mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl -0.07 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | COOKE MIDDLE SCHOOL AFTERSCHOOL RECREATION PROGRAM 


PHYSICAL ACTIVITY/HEALTH 


1+ physical activity 
hours/week 


8 8 Grades 5-8 220 0.10 (0.14) 


1+ strength training 


Tier Ill Tier Ill Grades 5-8 220 0.52 (0.14) * 
hours/week 


Average effect size for Physical Activity/Health outcomes at Tiers I-lIll | os 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


Stanford Achievement 
Test 9 - reading § § Grades 7-8 129 0.10 (0.18) 


comprehension 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III | ow | 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCHOOL ENGAGEMENT 


1+ homework hours/ 


Tier Ill Tier Ill Grades 5-8 220 0.49 (0.14) * 
week 


Average effect size for School Engagement outcomes at Tiers I-Ill | oo 


* Statistically significant at p<.05 


CITATION: Lauver, S. C. (2002). Assessing the benefits of an after-school program for urban youth: An impact and process evaluation (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3043903) 
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MULTICOMPONENT PROGRAMS 


Texas After School Initiative 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Texas After School Initiative, offered from 1999-2003, Location 
aimed to promote academic achievement and decrease LOCALE REGION 
involvement in the juvenile justice system for at-risk 10-to 
14-year-old students throughout the state of Texas. The 
programs were required to offer academic support linked to 


the state standards and include character and citizenship edu- 


cation as well as parent and/or mentor involvement. Programs 


were offered four days per week for two hours each day. 


Rural, Suburban, Urban Southwest 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 6-8 Low-Achieving Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Attendance & Enrollment 


OUTCOMES THAT DID NOT MEET TIER I-IIl STANDARDS FOR 
CAUSALITY: Mathematics Achievement and Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, School Engagement, Science Achievement, and Social & 24% Black 
Emotional Competencies 


62% Low-Income 54% Latino 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: None reported SETTING: School site 
BEST PRACTICES USED: Active learning, positive DELIVERY FORMAT: Group and one-to-one formats 


adult-student relationships, family engagement STAFFING: Certified teachers, volunteers 


DURATION: 4 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Ongoing support 
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MULTICOMPONENT PROGRAMS | TEXAS AFTER SCHOOL INITIATIVE 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of a statewide afterschool initiative in Texas serving students in grades 6-8. Outcomes for 
students who participated in the program were compared to outcomes for a matched group of students who did not participate. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 
activities. The study examines the impact of the program from 1999-2000 through 2002-03. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome Group 


Without site and With site and students (Standard error) 
sample criteria sample criteria 


peel ce Tier II Tier II sae 7 14,334 0.10 (0.02)* 
Senn hee Tier II Tier II Se i 14,334 0.07 (0.02)* 
aprtnerese pone Tier Il Tier Il oe Ly 14,334 0.05 (0.02)* 
ae cee Tier II Tier II ae, _— 32,621 0.12 (0.01)* 
carats ae Tier II Tier II oo 7 32,621 0.09 (0.01)* 
ee. fe Tier Il Tier I ae le 38,802 0.13 (0.01)" 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill | con 


*Statistically significant at p<.05 


CITATION: Shapley, K., Vicknair, K., Sheehan, D., Pieper, A., Jepson, D., & Sturges, K. (2004). Texas study of students at risk: Efficacy of grants supporting 
academic success from elementary through high school. Austin, TX: Texas Center for Educational Research. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers (St. Louis, Missouri) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Missouri’s 21st Century Community Learning Centers, funded Location 


by the US Department of Education, aimed to improve math LOCALE REGION 


and reading achievement of low-income students. Programs in 
this study took place in St. Louis during 1998-99 and targeted 
elementary and middle school students. Programs offered a 
range of activities including tutoring, activities designed to 
address social and behavioral issues, and recreational activi- 


ties. The duration of the programs is not reported. Wiklwast 


Participants 


CLASS GRADE TARGET STUDENTS 


Elementary & Low-Income 
EFFECTIVENESS AT A GLANCE Middle School 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Other Achievement, Physical Activity/Health, 


; F Not Reported Not Reported 
Promotion & Graduation, Reading/ELA Achievement, Attendance & 
Enrollment, School Engagement, Science Achievement, and Social 
& Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Sports, tutoring, SETTING: School site 


ial skill iviti ti 
social skills activities, recreation DELIVERY FORMAT: Group format 


: Not 
BEST PRACTICES USED: Not reported STAFFING: Program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (ST. LOUIS, MISSOURI) 


Study Details 


STUDY DESCRIPTION 

The quasi-experiment estimates the impact of the 21st Century Community Learning Centers program in in seven St. Louis schools 
during 1998-99. Comparison group students attended the same schools but did not participate in the program. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : . oie Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
District-developed | an 
math standardized Tier Il Tier Ill eee 278 0.62 (0.15)* 


middle 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl | os 


assessment 


* Statistically significant at p<.05 


CITATION: Nance, E. E., Moore, D. H., & Lewis, C. (1999). 21st Century Community Learning Centers: Do they affect student achievement? Community 
Education Journal, 27(1-2), 7-11. 
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MULTI-STUDY PROGRAM SUMMARY 


The After-School Corporation (PreK-Grade 8) 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


The After-School Corporation programs aimed 

to promote academic learning and healthy 
development as well as reduce anti-social behavior 
for children in New York City. Programs were 
school-based but supported by community orga- 
nizations. Activities varied but included academic 
support, arts and culture, physical activities, health 
education, community service, and technology. 
Older youth also received college and career 
preparation, internships, peer counseling, and 


violence prevention. 


NOTE: A separate program summary of the After- 
school Corporation for the middle grades only is 
also included in this evidence guide (see page 198). 


Positive Effect 


Overall, the studies found at least one improved 


outcome with no overriding contrary evidence for 


Attendance & Enrollment 


Mixed Effects 


Overall, the studies found a mix of improved and 
null or negative outcomes for 


Mathematics Achievement 


No Effect 


Overall, the studies found neither improved nor 
negative outcomes for 


Reading/ELA Achievement 


This review of The After-School Corporation (PreK-Grade 8) identified two studies of this program. The 
findings from each of these studies are presented individually on pages 215 — 220 of this evidence guide. 


- Reisner, E. R., White, R. N., Russell, C. A., & Birmingham, J. (2004). Building quality, scale, and effectiveness in 


afterschool programs: Summary report of the TASC evaluation. Washington, DC: Policy Studies Associates, Inc. 


(Note: Two studies are included in this manuscript). 


« White, R. N., Reisner, E. R., Welsh, M., & Russell, C. A. (2001). Patterns of student-level change linked to TASC 


participation, based on TASC projects in year 2. Washington, DC: Policy Studies Associates, Inc. 
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The After-School Corporation (PreK—Grade 8 — 1998-2004) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After-School Corporation programs, offered in 1998-2004, Location 


aimed to promote academic learning and healthy development LOCALE REGION 
and reduce anti-social behavior for children in New York City. 
Programs were school-based but supported by community 
organizations. Activities varied but included academic support, 
arts and culture, physical activities, health education, commu- 


nity service, and technology. The programs were offered for 


the full school year, five days per week, three hours each day. Nodieact 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades PreK-8 Low-Income 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Attendance & Enrollment 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement 7 7 

: : : 92% Low-Income 49% Latino 7% Asian 
Physical Activity/Health, Promotion & Graduation, Reading/ELA . : : 
Achievement, School Engagement, Science Achievement, and 39% Black 5% White 
Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic enrichment, arts, health SETTING: School site 


education, sports/recreation, other enrichment 
aoe ; DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Sequenced, active learning, 
positive adult-student relationships, family engagement, 
respect for student cultures DURATION: Full school year, 5 days/week, 3 hours/day 


STAFFING: Certified teachers, program staff 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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MULTI-STUDY PROGRAM SUMMARY THE AFTER-SCHOOL CORPORATION (PREK-GRADE 8) | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of The After-School Corporation (TASC) programs for students in grades PreK-8 in New 
York City. The comparison group is comprised of students at schools in the study implementing a TASC program who did not 
participate in the program. The comparison students experienced “business as usual,” which could have included other school- and 
community- sponsored afterschool activities. The study was conducted in 1998-2004. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 


Number of Effect size 
Outcome : ; a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Be Cee Tier Ill Tier Il Grades 77768 0.05 (0.01) * 
days attended Prek-8 ; : . 


Average effect size for Attendance & Enrollment outcomes at Tiers I-IIl 


* Statistically significant at p<.05 


CITATION: Reisner, E. R., White, R. N., Russell, C. A., & Birmingham, J. (2004). Building quality, scale, and effectiveness in afterschool programs: 
Summary report of the TASC evaluation. Washington, DC: Policy Studies Associates, Inc. 
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MULTI-STUDY PROGRAM SUMMARY THE AFTER-SCHOOL CORPORATION (PREK-GRADE 8) | SUMMARY #2 


The After-School Corporation (PreK—Grade 8 — 1998-2004) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After-School Corporation programs, offered in 1998-2004, Location 


aimed to promote academic learning and healthy development LOCALE REGION 
and reduce anti-social behavior for children in New York City. 
Programs were school-based but supported by community 
organizations. Activities varied but included academic support, 
arts and culture, physical activities, health education, commu- 


nity service, and technology. The programs were offered for 


the full school year, five days per week, three hours each day. Nodieact 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades PreK-8 Low-Income 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


NOT MEAS UEED FOR THIS INFERNENAAON: General SHEE? 93% Low-Income 49% Latino 7% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & 

Graduation, Reading/ELA Achievement, School Engagement, Science 39% Black 5% White 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic enrichment, arts, health SETTING: School site 


ion rts/recreation, other enrichmen 
education, sports/recreation, other enrichment DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Sequenced, active learning, 
positive adult-student relationships, family engagement, 
respect for student cultures DURATION: Full school year, 5 days/week, 3 hours/day 


STAFFING: Certified teachers, program staff 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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MULTI-STUDY PROGRAM SUMMARY THE AFTER-SCHOOL CORPORATION (PREK-GRADE 8) | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of The After-School Corporation (TASC) programs for students in grades 3-8 in New York 
City. The comparison group is comprised of schools in the study implementing a TASC program but who did not participate in the 
program. The comparison students experienced “business as usual,” which could have included other school- and community- 
sponsored after-school activities. The study was conducted in 1998-04. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; eae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percent of school 7 ‘ 
Tier Ill Tier III Grades 3-8 0.08 * 
days attended 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 0.08 


* Statistically significant at p<.05 


CITATION: Reisner, E. R., White, R. N., Russell, C. A., & Birmingham, J. (2004). Building quality, scale, and effectiveness in afterschool programs: 
Summary report of the TASC evaluation. Washington, DC: Policy Studies Associates, Inc. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 218 


MULTI-STUDY PROGRAM SUMMARY THE AFTERSCHOOL CORPORATION (PREK-GRADE 8) | SUMMARY #3 


The After-School Corporation (Grades 3-8 — 1999-2000) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After-School Corporation programs, offered in 1999-2000, Location 


aimed to promote academic learning and healthy development LOCALE REGION 
and reduce anti-social behavior for children in New York City. 
Programs were school-based but supported by community 
organizations. Activities varied but included academic support, 
arts and culture, physical activities, health education, commu- 


nity service, and technology. The programs were offered for 


the full school year, five days per week, three hours each day. 


Northeast 
Participants 
EFFECTIVENESS AT A GLANCE 
CLASS GRADE TARGET STUDENTS 
Grades 3-8 Low-Income 
Negative Effect 
The study found at least one negative outcome with no 
overriding contrary evidence for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mathematics Achievement 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & 


0, = 9 ‘ 9 ‘ 
Graduation, Attendance & Enrollment, School Engagement, Science 93% Low-Income 46% Latino 8% Asian 
Achievement, and Social & Emotional Competencies 39% Black 6% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Academic enrichment, arts, health SETTING: School site 


ion rts/recreation, other enrichment 
education: SPOrsit eh eainn, ether ennenme DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Sequenced, active learning, 
positive adult-student relationships, family engagement, 
respect for student cultures DURATION: Full school year, 5 days/week, 3 hours/day 


STAFFING: Certified teachers, program staff 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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MULTI-STUDY PROGRAM SUMMARY THE AFTERSCHOOL CORPORATION (PREK-GRADE 8) | SUMMARY #3 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the After-School Corporation (TASC) programs for students in grades 3-8 in New York 
City. The comparison group is comprised of schools in the study implementing a TASC program but who did not participate in the 
program. The comparison students experienced “business as usual,” which could have included other school- and community-spon- 
sored afterschool activities. The study was conducted in 1999-2000. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ea Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


New York State 
mathematics : 
assessments (grades oo co sass 23,009 -0.05 (0.01) * 
4, 8) and aligned 
tests for other grades 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ‘ ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


New York State 
English language arts : 
assessments (grades 4, oo oo ee 22,128 0.00 (0.01) 
8) and aligned tests for 
other grades 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


CITATION: White, R. N., Reisner, E. R., Welsh, M., & Russell, C. A. (2001). Patterns of student-level change linked to TASC participation, based on TASC 
projects in year 2. Washington, DC: Policy Studies Associates, Inc. 
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MULTICOMPONENT PROGRAMS 


Cool Girls 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Cool Girls, offered at eight schools in the Southeast during Location 
2005-06, was a school-based afterschool program designed to LOCALE REGION 
increase the psychological and social resources of low-income 
girls in grades 4-8. The program had three key components: 
Girls Club, which focused on life skills; Cool Scholars, which 
focused on academic skill-building; and Cool Sisters, a one- 


to-one mentoring program. In addition, the program included 


weekend workshops, special events, field trips, and a summer 


Southeast 
program. The program was offered for the full school year, one 
day per week. 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 4-8 Low-Income Girls 
Positive Effect 
The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 


Physical Activity/Health 


No Effect 


The study found neither improved nor negative outcomes for 


School Engagement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 80% Low-Income 89% Black 2% Asian 
ment, Mathematics Achievement, Other Achievement, Promotion & 

Graduation, Reading/ELA Achievement, Attendance & Enrollment, 4% Latino 4% Other 
Science Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Tutoring, homework help, SETTING: School site 


Icohol 
drug/aleshobpreventen)mentenng DELIVERY FORMAT: Group and one-to-one formats 


BEST PRACTICES USED: Positive adult-student 


relationships, family engagement, respect for 
student cultures DURATION: Full school year, 1 day/week 


STAFFING: Program staff 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | COOL GIRLS 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment compared outcomes for students who participated in Cool Girls to outcomes for a matched group of students 
who did not participate. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. The study was conducted at eight schools during the 2005-06 school year. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome : ; oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Number of fruits and 
vegetables eaten in oo co Grades 4-8 175 0.19 (0.15) 
past day (self-report) 


Number of physical 
activities in the past Tier Il Tier Ill Grades 4-8 175 0.53 (0.15)* 
week (self-report) 


Average effect size for Physical Activity/Health outcomes at Tiers I-III 


* Statistically significant at p<.05 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


SCHOOL ENGAGEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : é eee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Internal motivation 
adapted from existin 
Sele : 00 00 Grades 4-8 175 0.15 (015) 


measures (Harter, 1982; 
Ryan & Connell, 1989)) 


Number of days per 
week of studying after co co Grades 4-8 175 0.17 (0.15) 
school (self-report) 


Average effect size for School Engagement outcomes at Tiers I-Ill 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Kuperminc, G. P., Thomason, J., DiMeo, M., & Broomfield-Massey, K. (2011). Cool Girls, inc.: Promoting the positive development of urban 
preadolescent and early adolescent girls. Journal of Primary Prevention, 32(3-4), 171-183. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers 


(Philadelphia - High School) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


These 21st Century Community Learning Centers programs, Location 


offered in Philadelphia in 2012-13, provided academic support LOCALE REGION 
and other enrichment activities such as athletics, art and music, 
recreation, and career and technical education. The aim was 

to improve academic and behavioral outcomes of low-income 
youth attending high-poverty, low-performing schools. The 


programs were offered for the full school year, five days per 


week, for up to three hours per day. 


Northeast 
Participants 
Grades 9-12 Low-Income 
Positive Effect 
The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 


Promotion & Graduation 


No Effect 


The study found neither improved nor negative outcomes for 


Attendance & Enrollment 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical Activity/ 
Health, Reading/ELA Achievement, School Engagement, Science 28% Latino 
Achievement, and Social & Emotional Competencies 


73% Low-Income 66% Black 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Arts, sports, tutoring, homework SETTING: School site 


ieee 
Me lPucteekie Scinatles DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Not reported STAFFING: Program staff 
DURATION: Full school year, 5 days/week, 2.5-3 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (PHILADELPHIA - HIGH SCHOOL) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool programs in Philadelphia offered by six 21st Century Community Learning 
Center grantees. Outcomes for participants who attended at least 30 days of the program are compared to outcomes for students who 
did not participate or who participated for fewer than 30 days. The comparison students experienced “business as usual,” which could 
have included other school- and community-sponsored afterschool activities. The study was conducted during 2012-13. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
10 or more unexcused . nn nn 
§ § High school Pannen Cannot 
absences determine determine 


: : Cc t 
Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill Bae 


determine 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


PROMOTION & GRADUATION 


ESSA evidence tier 
Number of Effect size 
Outcome 5 i Tass Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
‘ : : : F Cannot Cannot 
Earned English credit : : 
d English cred Tier Ill Tier Ill High school determine determine 
Earned mathematics : ' F Cannot Cannot 
credit Tent a Bligh: schne@ol determine determine* 
; : é Cannot Cannot 
Earned science credit 5 
= i ; 8 8 Bigh:echoo! determine determine 


Average effect size for Promotion & Graduation outcomes at Tiers I-IIl Saangt 


determine 


* Statistically significant at p<.05 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Gao, J., Hallar, B., & Hartmann, T. A. (2014). A snapshot of OST programs in Philadelphia: An evaluation of eleven 21st Century Community 
Learning Center grantees. Philadelphia, PA: Research for Action. 


Hartmann, T. A., Gao, J., Kumar, A., & Edmunds, K. (2013). A snapshot of OST programming in Philadelphia: An evaluation of six 21st Century Community 
Learning Center grantees. Philadelphia, PA: Research for Action. 
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MULTICOMPONENT PROGRAMS 


After School Safety and Enrichment For Teens 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After School Safety and Enrichment for Teens (ASSETs) Location 
program, offered from 2007-10 and funded by the State of 
California, served students in over 300 schools in the state. 


LOCALE REGION 


Sites offered academic and non-academic enrichment. The 
most commonly offered activities were academic enrichment, 


arts/music, homework assistance, physical fitness/sports, 


recreation, and tutoring. Programs were offered for the full 
school year, five days per week, three hours each day. 


Suburban, Rural, Urban West 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 9-11 Low-Income 


Positive Effect 


The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 
Attendance & Enrollment 
Mathematics Achievement 


Reading/ELA Achievement 


Physical Activity/Health 


; 78% Low-Income 73% Latino 8% White 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Promotion & Graduation, School Engage- 10% Asian 2% Other 
ment, Science Achievement, and Social & Emotional Competencies 8% Black 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic enrichment, arts, SETTING: School site 


health ti i th ich 
ealth education, sports/recreation, other enrichment DELIVERY FORMAT: Group formats 


BEST PRACTICES USED: Positive adult-student 


STAFFING: Certified teachers, program staff 
relationships, family engagement 


DURATION: Full school year, 5 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL SAFETY AND ENRICHMENT FOR TEENS 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact of After School Safety and Enrichment for Teens on students who attended the program for 


at least one day during a given school year. The comparison group is comprised of students who never participated in the program. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 
activities. The study was conducted in 2007-10. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : A a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percentage of : : * 
Tier Ill Tier Ill Grades 9-11 65,616 0.14 (0.01) 


days attended 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill pow 


* Statistically significant at p<.05 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ' Pe, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


California High School 
Exit Examination Tier Ill Tier Ill Grades 9-11 76,294 0.08 (0.01)* 


(CAHSEE) mathematics 


California Standards ; ‘ 0.02 
Tests (CST) mathematics Tier Ill Tier Ill Grades 9-11 212,412 (0.004)* 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl | 00s 


* Statistically significant at p<.05 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL SAFETY AND ENRICHMENT FOR TEENS 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 

Outcome : ; ee Group 

Without site and With site and students (Standard error) 
sample criteria sample criteria 

Aerobic Capacity Tier Ill Tier Ill Grades 9-11 70,278 0.17 (0.01)* 
Body Composition Tier Ill Tier Ill Grades 9-11 70,278 0.16 (0.01)* 
Abdominal Strength Tier Ill Tier Ill Grades 9-11 70,278 0.09 (0.01)* 
Trunk Strength Tier Ill Tier Ill Grades 9-11 70,278 0.14 (0.01)* 
Upper Body Strength Tier Ill Tier Ill Grades 9-11 70,278 0.16 (0.01)* 
Flexibility Tier Ill Tier Ill Grades 9-11 70,278 0.15 (0.01)* 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill pow 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; e nae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California High School 
Exit Examination 7 7 
Tier Ill Tier Ill Grades 9-11 76,380 0.06 (0.01)* 


(CAHSEE) English 
language arts 


California Standards 
Tests (CST) English Tier III Tier Ill Grades 9-11 224,839 
language arts 


0.03 
(0.004)* 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III | 00s | 


* Statistically significant at p<.05 


CITATION: Huang, D., & Wang, J. (2012). Independent statewide evaluation of high school after school programs: May 1, 2008-December 31, 2011 
(CRESST Report). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation, 
Graduate School of Education & Information Studies, University of California, Los Angeles. 
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MULTICOMPONENT PROGRAMS 


The Afterschool Corporation (High School) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Afterschool Corporation (TASC) programs, offered in Location 
New York City from 1998-2004, aimed to promote academic LOCALE REGION 
learning and healthy development as well as reduce anti-social 
behavior for in New York City. Programs were school-based but 
supported by community organizations. Activities varied but 

included academic support, arts and culture, physical activities, 
health education, community service, and technology. Youth 


also received college and career preparation, internships, peer 


Northeast 
counseling, and violence prevention. 
Participants 
= 4 en BAVA ae Wed VN Led = CLASS GRADE TARGET STUDENTS 
Grades 9-12 Low-Income 
Positive Effect 
The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 


Attendance & Enrollment 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 

ment, Mathematics Achievement, Other Achievement, Physical 76% L r : 
ow-Income 45% Latino 12% White 

Activity/Health, Promotion & Graduation, Reading/ELA Achievement, . . 7 

School Engagement, Science Achievement, and Social & Emotional 32% Black 11% Asian 

Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic enrichment, arts, health SETTING: School site 


ion rts/recreation, other enrichment 
Sducabion: SPOrsi eh Salon, einer ennenme DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Sequenced, active learning, 
positive adult-student relationships, family engagement, 
respect for student cultures DURATION: Full school year, 5 days/week, 3 hours/day 


STAFFING: Certified teachers, program staff 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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MULTICOMPONENT PROGRAMS | THE AFTERSCHOOL CORPORATION (HIGH SCHOOL) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the Afterschool Corporation (TASC) programs for students in grades 9-12 in New York 
City. The comparison group is comprised of students at schools in the study implementing a TASC program who did not participate in 
the program. The comparison students experienced “business as usual,” which could have included other school and community- 
sponsored afterschool activities. The study was conducted in 1998-2004. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . “ rage Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percent of school 5 
Tier Ill Tier Ill Grades 9-12 6,605 0.38 (0.03) * 


days attended 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


CITATION: Reisner, E. R., White, R. N., Russell, C. A., & Birmingham, J. (2004). Building quality, scale, and effectiveness in afterschool programs: 
Summary report of the TASC evaluation. Washington, DC: Policy Studies Associates, Inc. 

NOTE: A second study (Birmingham & White, 2005) examined the impacts of TASC for a partially overlapping sample of high school students. The study 
also had one outcome at Tier III (attendance), with the same reported effect size and standard error for the attendance outcome as in the Reisner, White, 
Russell, & Birmingham (2004) study. We count these as two separate studies but, for parsimony, report only the finding from the 2004 study in this 
program summary. The program’s ESSA rating is the same regardless of whether we combine results from both studies or present one study only. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers (Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This is a national study of the impact of 21st Century ; 
Community Learning Centers in 2000-02. Programs served Location 
low-income students in school districts around the country, LOCALE REGION 
including those in rural and urban areas, and aimed to provide 
academic support, recreation, and enrichment activities. 
Programs typically operated in school buildings but were 


sometimes run by community organizations. Programs offered 


a range of programming, including homework help, academic 
activities, athletics, arts and cultural activities, life skills training, Suburban, Rural, Urban All regions 
and unstructured recreational time. Programs were offered for 


the full school year, five days per week, three hours each day. 


Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 


Grades K-6 Low-Income 


No Effect 


The study found neither improved nor negative outcomes for 


Attendance & Enrollment SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Mathematics Achievement 
Reading/ELA Achievement 


Science Achievement 


Negative Effects 


The study found at least one negative outcome with no 


overriding contrary evidence for 
Most schools had more than 67% Black 10% White 


Science Achievement 50% low-income students 19% Latino 2% Other 


School Engagement 


Other Achievement 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Physical Activity/Health, and Promotion & Graduation 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Academic tutoring, homework SETTING: School site 


ti ial devel 
help, arts, sports/recreation, social development DELIVERY FORMAT: Group format 


Beet ERA GASES ane Pirepolied STAFFING: Certified teachers, program staff 
DURATION: Full school year, 5 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 
This randomized controlled trial estimates the impact of afterschool programs offered to students in grades K-6 at 26 sites in 12 school 


districts. Outcomes for students who were offered the opportunity to participate in an afterschool program are compared to outcomes 
for students who did not receive this offer. The comparison students experienced “business as usual,” which could have included other 
school- and community-sponsored afterschool activities. The study was conducted in 2000-01 and 2001-02. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome , g wen Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school 
) ) Grades K-6 1,551 -0.03 (0.05) 
days absent 
Number of school Cannot 
y) ) Grades K-6 1,551 determing 


days late 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill | cos 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; , re Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Mathematics 
) ) Grades K-6 1,551 -0.06 (0.05) 


course grade 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (ELEMENTARY) 


READING/ELA ACHIEVEMENT 


English language arts 
course grade 


9 9 Grades K-6 1,551 0.03 (0.05) 


Stanford Achievement 
Test 9 — reading ) ) Grades K-6 1,748 0.01 (0.05) 


comprehension 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III | 002 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


OTHER ACHIEVEMENT 


Social studies/history 
course grade 


) ) Grades K-6 1,551 -0.11 (0.05) * 


Average effect size for Other Achievement outcomes at Tiers I-Ill pon 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (ELEMENTARY) 


SCHOOL ENGAGEMENT 


Achieves at above- 
average or very high ) » Grades K-6 1,778 -0.19 (0.05) * 
level (teacher report) 


Comes to school 
prepared and ready to ) > Grades K-6 1,778 -0.10 (0.05) * 
learn (teacher report) 


Completes assignments 
to teacher’s satisfaction > ) Grades K-6 1,778 -0.04 (0.05) 
(teacher report) 


Student effort C : 
: anno 
composite ) ) Grades K-6 1,778 detatming 
(teacher report) 
eee ea ) ) Grades K-6 1778 -0.14 (0.05) * 


reading (teacher report) 


Average effect size for School Engagement outcomes at Tiers I-lll | om 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


SCIENCE ACHIEVEMENT 


Science course grade > ) Grades K-6 1,551 -0.02 (0.05) 


Average effect size for Science Achievement outcomes at Tiers I-III | oo 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (ELEMENTARY) 


SOCIAL & EMOTIONAL COMPETENCIES 


Percent of students 
whose teachers report 
that the student gets 
along well with others 


) ) Grades K-6 1,778 -0.19 (0.05) * 


Average effect size for Social & Emotional Competencies outcomes at Tiers I-Ill 


* Statistically significant at p<.05 
) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATIONS: Dynarski, M., Moore, M., Mullens, J., Gleason, P., James-Burdumy, S., Rosenberg, L., .. ., Deke, J. (2003). When schools stay open late: The 
national evaluation of the 21st Century Community Learning Centers program. First year findings (Report No. ED473230). Princeton, NJ: Mathematica 
Policy Research. 


Dynarski, M., Moore, M., James-Burdumy, S., Rosenberg, L., Deke, J., & Mansfield, W. (2004). When schools stay open late: The national evaluation of the 
21st Century Community Learning Centers program. New findings (MPR Report No. 8658-800/8720-700). Princeton, NJ: Mathematica Policy Research. 


James-Burdumy, S., Dynarski, M., & Deke, J. (2007). When elementary schools stay open late: Results from the national evaluation of the 21st Centruy 
Community Learning Centers program. Educational Evaluation and Policy Analysis, 29(4), 296-318. 


James-Burdumy, S., Dynarski, M., & Deke, J. (2008). Afterschool program effects on behavior: Results from the 21st Century Community Learning Centers 
program national evaluation. Economic Inquiry, 46(1), 13-18. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers (Louisiana) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


These 21st Century Community Learning Centers programs, Location 

funded by the U.S. Department of Education, were offered in LOCALE REGION 
four Louisiana school districts, two urban and two rural (Baton 
Rouge, New Orleans, Grant Parish, and Bienville Parish) during 
2003-04. The study reported no details about the program 
components or duration. 


Rural, Urban Southeast 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 3-5 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


General Achievement 


NOT MEASURED FOR THIS INTERVENTION: Mathematics 
Achievement, Other Achievement, Physical Activity/Health, Promotion 


0, i 0, 
& Graduation, Reading/ELA Achievement, Attendance & Enrollment, 80% Low-Income 59% Black 
School Engagement, Science Achievement, and Social & Emotional 40% White 
Competencies 1% Asian 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: None reported SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (LOUISIANA) 


Study Details 


STUDY DESCRIPTION 

The study uses a quasi-experimental design to estimate the impact of the program on students in four school districts, two urban and 
two rural, in Louisiana (Baton Rouge, New Orleans, Grant Parish, and Bienville Parish). Outcomes for participants are compared to 
outcomes for non-participants. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. The study was conducted in 2003-04. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . “ ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
lowa Test of Basic 
8 8 Grades 3-5 192 0.10 (0.07) 


Skills - core 


Average effect size for General Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Jenner, E., & Jenner, L. W. (2007). Results from a first-year evaluation of academic impacts of an afterschool program for at-risk students. 
Journal of Education for Students Placed at Risk, 12, 213-237. 
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MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers (New Hampshire) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


These 21st Century Community Learning Centers programs, Location 
funded by the US Department of Education, were offered in LOCALE REGION 
five elementary schools in New Hampshire in 2008-13. 21st 
Century Community Learning Centers aimed to improve the 
math and reading achievement of low-income students. The 
afterschool program activities included tutoring, homework 
support, drug and violence prevention, technology education, 


art/music, recreation, character education, and counseling. 


Northeast 
The program was offered for the full school year, five days per 
week, three hours each day. 
Participants 

EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 

Grades 1-5 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


; : ae s 71% Low-Income 66% White 
ment, Other Achievement, Physical Activity/Health, Promotion & . 
Graduation, Attendance & Enrollment, School Engagement, Science 25% Latino 
Achievement, and Social & Emotional Competencies 10% Black 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Tutoring, homework help, SETTING: School site 


iol ion, technol i 
drug and violence prevention, technology education, art/ DELIVERY FORMAT: Group format 


music, recreation, character education, counseling 


STAFFING: Program staff 
BEST PRACTICES USED: Positive adult-student 
relationships, family engagement DURATION: Full school year, 5 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | 21ST CENTURY COMMUNITY LEARNING CENTERS (NEW HAMPSHIRE) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment compared outcomes for students who participated in afterschool programs funded through the 21st Century 
Community Learning Centers program during 2008-13. Study participants were from five low-income elementary schools in New 
Hampshire. Outcomes for students who participated in the programs were compared to outcomes for non-participants. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i ‘ ane Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
New England 
Common Assessments oo co Grades 1-5 1,955 0.00 (0.05) 


Program - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; on Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
New England 
Common Assessments co co Grades 1-5 1,955 0.00 (0.05) 


Program - reading 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Scarpati, J. (2017). Are 21st Century after school programs an effective academic intervention for elementary school students attending 
high-poverty schools? (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 10273768) 
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MULTI-STUDY PROGRAM SUMMARY 


21st Century Community Learning Centers 
(West Virginia — Elementary) 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


West Virginia’s 21st Century Community Learning No Effect 


Centers, offered from 2012-2015, aimed to improve Overall, the studies found neither improved nor 


math and reading achievement of low-income negative outcomes for 


students. Programs served K-12 students through- Mathematics Achievement 


out the state and were operated by 33 separate 


programs. Afterschool programs offered a range of Reading/ELA Achievement 


activities including STEM programming, vocational 
enrichment, and professional presenters on a range 
of topics. A summer camp was also offered. 


This review of 21st Century Community Learning Centers (West Virginia — Elementary) identified 
three studies of this program (two manuscripts describe the same study). The findings from each of 
these studies are presented individually on pages 240 — 245 of this evidence guide. 


« White, L. J. (2014). 27st Century Community Learning Centers: A descriptive evaluation for 2012-13. Charleston, WV: 
West Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


« White, L., & Hixson N. (2014). 21st Century Community Learning Centers 2013: A quasi-experimental investigation 
of program impacts on student achievement in mathematics and reading/language arts. Charleston, WV: West 


Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


« White, L. J., Hammer, P. C., & Whisman, A. (2015). 21st Century Community Learning Centers, 2014-2015: A 
descriptive evaluation. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Assessment. 


« White, L. J., & Whisman, A. (2015). 21st Century Community Learning Centers 2014: A quasi-experimental 
investigation of program impacts on student achievement in mathematics and reading/language arts. 


Charleston, WV: West Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — ELEMENTARY) | SUMMARY #1 


21st Century Community Learning Centers 
(West Virginia - Elementary - 2012-13) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2012-13, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 4-5 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion and Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year, attendance range=2-93 days 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — ELEMENTARY) | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 4-5 of participating in 21st Century Community Learning 
Center programs during 2012-13. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ena Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 4 1,014 0.02 (0.06) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 5 930 0.02 (0.07) 


Test 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; aot Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 4 1,014 0.12 (0.06) 
language arts 
West Virginia 
Educational Standards 
Test 2 - English § § Grade 5 930 0.02 (0.07) 
language arts 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.07 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: White, L., & Hixson N. (2014). 21st Century Community Learning Centers 2013: A quasi-experimental investigation of program impacts on 
student achievement in mathematics and reading/language arts. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Research. 


White, L. J. (2014). 27st Century Community Learning Centers: A descriptive evaluation for 2012-13. Charleston, WV: West Virginia Department of 
Education, Division of Teaching and Learning, Office of Research. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — ELEMENTARY) | SUMMARY #2 


21st Century Community Learning Centers 
(West Virginia - Elementary - 2013-14) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2013-14, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 4-5 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion and Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — ELEMENTARY) | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 4-5 of participating in 21st Century Community Learning 
Center programs during 2013-14. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ena Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 4 904 0.03 (0.07) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 5 846 0.05 (0.07) 


Test 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; aot Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 
Test 2 - English g g Grade 4 904 0.08 (0.07) 
language arts 
West Virginia 
Educational Standards 
Test 2 - English g g Grade 5 930 0.08 (0.07) 
language arts 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.08 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: White, L. J., & Whisman, A. (2015). 21st Century Community Learning Centers 2014: A quasi-experimental investigation of program impacts 
on student achievement in mathematics and reading/language arts. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Research. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — ELEMENTARY) | SUMMARY #3 


21st Century Community Learning Centers 
(West Virginia - Elementary - 2014-15) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2014-15, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 4-5 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion and Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — ELEMENTARY) | SUMMARY #3 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 4-5 of participating in 21st Century Community Learning 
Center programs during 2014-15. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : ; ena Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 4 578 0.03 (0.07) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 5 312 0.05 (0.07) 


Test 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-lll 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; aos Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 
Test 2 - English g g Grade 4 578 0.08 (0.07) 
language arts 
West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 5 312 0.08 (0.07) 
language arts 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.08 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: White, L. J., Hammer, P. C., & Whisman, A. (2015). 27st Century Community Learning Centers, 2014-2015: A descriptive evaluation. 
Charleston, WV: West Virginia Department of Education, Division of Teaching and Learning, Office of Assessment. 
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MULTICOMPONENT PROGRAMS 


After School Education and Safety 


(San Francisco — Elementary) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After School Education and Safety (ASE) initiative of the Location 
San Francisco Unified School District, offered in 31 schools 


LOCALE REGION 
in 2005-06, aimed to improve student achievement and 


decrease involvement in unhealthy behaviors through provid- 
ing safe spaces, expanded learning opportunities, and social 


service supports. ASE offered multicomponent programming 
that included three types of activities: education, enrichment/ 


recreation, and life skills or drug and alcohol prevention. The 


West 
programs were offered for the full school year. 
Participants 
= td en w AVA Se Wed WN ed = CLASS GRADE TARGET STUDENTS 
Grades K-5 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mathematics Achievement 


Reading/ELA Achievement «) @ 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, = 9 9 

ment, Other Achievement, Physical Activity/Health, Promotion & 79% Low-Income 67% Latino 8% Asian 
Graduation, Attendance & Enrollment, School Engagement, Science 13% White 2% Other 
Achievement, and Social & Emotional Competencies 11% Black 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Arts, sports, tutoring, drug/alcohol SETTING: School site 


revention, cl ial skills activiti 
prevention, clubs, social skills activities DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Family engagement 
ny eae STAFFING: Certified teachers, volunteers, program staff 
DURATION: Full school year, no other info provided 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL EDUCATION AND SAFETY (SAN FRANCISCO — ELEMENTARY) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on academic achievement and attendance of the After School Education and Safety 
Program for students in the elementary grades in San Francisco Unified School District. Outcomes for participants were compared to 
those for non-participants. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. The study was conducted in 31 schools during 2005-06. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ane Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Proficient level on | 
California Standards co oo pao 7,665 0.01 (0.02) 


Test - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; on Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Proficient level on 
California Standards Elementary 
: oo oo 7,654 0.01 (0.02 
Test - English grades ( ) 


language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Roberson, I. (2006). After school education and safety program (Final Evaluation Report). San Francisco, CA: San Francisco Unified School 
District, Program Evaluation and Research Department. 
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MULTICOMPONENT PROGRAMS 


Boston Elementary School Afterschool Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Boston Elementary School Afterschool Program, offered at Location 
one elementary school in 2005-06, was designed to com- LOCALE REGION 
plement and extend school day learning. The school-based 
program was staffed primarily by school day teachers and 
offered a range of academic and enrichment activities includ- 
ing architecture, chess, computers, crochet, dance, French, 


math club, music, and swimming. The program was offered for 


the full school year, five days per week, 3.75 hours each day. 


Northeast 
Participants 
Grades K-8 General Population 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 67% Low-lacomeé 37% White 20% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & 

Graduation, Attendance & Enrollment, School Engagement, Science 36% Black 6% Latino 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Academic enrichment, arts and SETTING: School site 


| ti 
Culture; sportsirecieston DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Active learning, 
family engagement 


STAFFING: Certified teachers 
DURATION: Full school year, 5 days/week, 3.75 hours/day 


IMPLEMENTATION SUPPORT: Ongoing support 
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MULTICOMPONENT PROGRAMS | BOSTON ELEMENTARY SCHOOL AFTERSCHOOL PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool program for students in grades 3-5 at one elementary school in Boston, 
Massachusetts. Outcomes for participants were compared to outcomes for non-participants attending the same school and in the same 
grades. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. The study was conducted in 2005-06. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome . ‘ an Group 
Without site and With site and students (Standard error) 


sample criteria sample criteria 


Massachusetts 
Comprehensive 
Assessment System 8 8 Grades K-8 162 0.08 (0.17) 
— mathematics — 
1-year participants 


Massachusetts 
Comprehensive 
Assessment System 8 8 Grades kK-8 132 0.15 (0.23) 
— mathematics — 
2-year participants 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i A wer Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Massachusetts 
Comprehensive Assess- 
ment System - English § 8 Grades K-8 162 0.02 (0.17) 
language arts — 
1-year participants 
Massachusetts 
Comprehensive Assess- 
ment System - English § 8 Grades kK-8 132 0.15 (0.23) 
language arts — 
2-year participants 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.07 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Moldow, E. (2007). Afterschool program activities and academic achievement: A study in one urban k—8 school (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3278159) 
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MULTICOMPONENT PROGRAMS 


Hope Center for Kids 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Hope Center for Kids, offered in Omaha, Nebraska during Location 
aimed to support the development of Christian faith as well as LOCALE REGION 


the academic success, high school graduation, post-secondary 
training, and employability of elementary school children who 
attended local public schools. The faith-based program offered 
homework help and dedicated homework time, study skills 
classes, mentoring, and faith-based activities (e.g., prayer and 


Bible study). The program was offered for the full school year, 


Midwest 
five days per week, 3.5 hours each day. The implementation 
year is not reported. 
Participants 

=f 4 en BAVA Se Wed eV red = CLASS GRADE TARGET STUDENTS 

Grades 2-5 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Attendance & Enrollment 
Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, a! 0, 
ment, Other Achievement, Physical Activity/Health, Promotion & 100% Low-Income 100% Black 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Homework help, mentoring, SETTING: Community site 


Itip| 
EON OS eae DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Positive adult-student 


relationships, family engagement, respect for 
student cultures DURATION: Full school year, 5 days/week, 3.5 hours/day 


STAFFING: Program staff 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | HOPE CENTER FOR KIDS 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment compared outcomes for students who attended an afterschool program at the Hope Center for Kids in Omaha, 
Nebraska to outcomes for non-participants. The comparison students experienced “business as usual,” which could have included other 
school- and community-sponsored afterschool activities. The implementation year is not reported. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ane Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Days absent 
e co co Grades 2-5 30 -0.34 (0.38) 


from school 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome ; ; ap Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
California Achievement 
co co Grades 2-5 30 -0.31 (0.38) 


Test - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-lll 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | HOPE CENTER FOR KIDS 


READING/ELA ACHIEVEMENT 


California Achievement 
Test - language 


oo oo Grades 2-5 30 -0.54 (0.39) 


California Achievement 


Test - reading . . Grades 2-5 30 -0.55 (0.39) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Yokley-Busby, S. (2013). The impact of attendance longevity in an after school program, designed to build intentional relationships and 
support academic success, on urban elementary students’ achievement, attendance, and school awards (Doctoral dissertation). Retrieved from 
ProQuest Dissertations and Theses Database (UMI No. 3601426) 
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MULTI-STUDY PROGRAM SUMMARY 


21st Century Community Learning Centers 


(West Virginia — Middle Grades) 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


West Virginia’s 21st Century Community Learning No Effect 


Centers aimed to improve math and reading Overall, the studies found neither improved nor 


achievement of low-income students. Programs negative outcomes for 


served K-12 students throughout the state and were Mathematics Achievement 


operated by 33 separate programs. Afterschool 


programs offered a range of activities including Reading/ELA Achievement 


STEM programming, vocational enrichment, and 
professional presenters on a range of topics. A 


summer camp was also offered. 


This review of 21st Century Community Learning Centers (West Virginia — Middle Grades) identified 
three studies of this program (two manuscripts report the same study). The findings from each of these 
studies are presented individually on pages 254 — 262 of this evidence guide. 


« White, L., & Hixson N. (2014). 21st Century Community Learning Centers 2013: A quasi-experimental investigation 
of program impacts on student achievement in mathematics and reading/language arts. Charleston, WV: West 


Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


« White, L. J. (2014). 27st Century Community Learning Centers: A descriptive evaluation for 2012-13. Charleston, WV: 
West Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


« White, L. J., Hammer, P. C., & Whisman, A. (2015). 21st Century Community Learning Centers, 2014-2015: A 
descriptive evaluation. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Assessment. 


« White, L. J., & Whisman, A. (2015). 21st Century Community Learning Centers 2014: A quasi-experimental 
investigation of program impacts on student achievement in mathematics and reading/language arts. Charleston, 


WV: West Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #1 


21st Century Community Learning Centers 


(West Virginia - Middle Grades - 2012-13) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2012-13, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 6-8 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion and Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year, attendance range=2-93 days 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 6-8 of participating in 21st Century Community Learning 
Center programs during 2012-13. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 6 700 0.01 (0.08) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 7 390 0.04 (0.10) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 8 352 -0.05 (0.11) 


Test 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #1 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome , ; en Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 6 700 0.01 (0.08) 


language arts 


West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 7 390 0.07 (0.10) 


language arts 


West Virginia 
Educational Standards 
Test 2 - English 
language arts 


§ § Grade 8 352 0.04 (0.11) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: White, L., & Hixson N. (2014). 21st Century Community Learning Centers 2013: A quasi-experimental investigation of program impacts on 
student achievement in mathematics and reading/language arts. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Research. 


White, L. J. (2014). 21st Century Community Learning Centers: A descriptive evaluation for 2012-13. Charleston, WV: West Virginia Department of 
Education, Division of Teaching and Learning, Office of Research. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #2 


21st Century Community Learning Centers 


(West Virginia - Middle Grades - 2013-14) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2013-14, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 6-8 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion and Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 6-8 of participating in 21st Century Community Learning 
Center programs during 2013-14. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 6 484 0.03 (0.09) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 7 362 0.04 (0.11) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 8 360 0.12 (0.11) 


Test 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #2 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; Sree Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 6 484 0.06 (0.09) 
language arts 
West Virginia 
Educational Standards 
Test 2 - English § § Grade 7 362 0.04 (0.11) 
language arts 
West Virginia 
Educational Standards 
§ g Grade 8 360 0.15 (0.11) 


Test 2 - English 
language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: White, L. J., & Whisman, A. (2015). 27st Century Community Learning Centers 2014: A quasi-experimental investigation of program impacts 
on student achievement in mathematics and reading/language arts. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Research. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #3 


21st Century Community Learning Centers 
(West Virginia - Middle Grades - 2014-15) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2014-15, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 6-8 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion and Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #3 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 6-8 of participating in 21st Century Community Learning 
Center programs during 2014-15. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; Prael Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 6 312 0.03 (0.09) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 7 166 0.04 (0.11) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 8 194 0.15 (0.11) 
Test 2 - mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 0.07 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — MIDDLE GRADES) | SUMMARY #3 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome A ; en Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 
Test 2 - English g g Grade 6 312 0.06 (0.09) 
language arts 
West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 7 166 0.04 (0.11) 
language arts 
West Virginia 
Educational Standards 
Test 2 - English cS} 8 Grade 8 194 0.12 (0.11) 
language arts 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 0.08 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: White, L. J., Hammer, P. C., & Whisman, A. (2015). 21st Century Community Learning Centers, 2014-2015: A descriptive evaluation. Charles- 
ton, WV: West Virginia Department of Education, Division of Teaching and Learning, Office of Assessment. 
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MULTICOMPONENT PROGRAMS 


After School Education and Safety (San Francisco — Middle Grades) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The After School Education and Safety (ASE) initiative of the Location 
San Francisco Unified School District, offered in 31 schools LOCALE REGION 
in 2005-06, aimed to improve student achievement and 

decrease involvement in unhealthy behaviors through provid- 


ing safe spaces, expanded learning opportunities and social 


service supports. ASE offered multicomponent programming 
that included three types of activities: education, enrichment/ 


recreation, and life skills or drug and alcohol prevention. The 


West 
programs were offered for the full school year. 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 6-8 Low-Income 

No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 

Mathematics Achievement 

Reading/ELA Achievement 
OUTCOMES THAT DID NOT MEET TIER I-IIl STANDARDS 
FOR CAUSALITY: Attendance & Enrollment Not measured for this 79% Low-Income 67% Latino 8% Asian 
intervention: General Achievement, Other Achievement, Physical 
Activity/Health, Promotion and Graduation, School Engagement, 13% White 2% Other 
Science Achievement, and Social & Emotional Competencies 11% Black 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Arts, sports, tutoring, drug/alcohol SETTING: School site 


revention, cl ial skills activiti 
prevention, clubs, social skills activities DELIVERY FORMAT: Not reported 


BEST PRACTICES USED: Family engagement 
ny eae STAFFING: Certified teachers, volunteers, program staff 
DURATION: Full school year, no other info provided 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTICOMPONENT PROGRAMS | AFTER SCHOOL EDUCATION AND SAFETY (SAN FRANCISCO — MIDDLE GRADES) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact on academic achievement and attendance of the After School Education and Safety 
Program for students in the elementary grades in San Francisco Unified School District. Outcomes for participants were compared to 
those for non-participants. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. The study was conducted in 31 schools during 2005-06. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ane Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Proficient level on A 
Middle 


California Standards co oo grades 838 0.00 (0.07) 
Test - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; ae Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 
Proficient level 

on California Middle 
oo oo -0.01 (0.07 

Standards Test - grades ooF 0.01 (0.07) 


English language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Roberson, I. (2006). After school education and safety program (Final Evaluation Report). San Francisco, CA: San Francisco Unified School 
District, Program Evaluation and Research Department. 
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MULTICOMPONENT PROGRAMS 


Afterschool Program with All Stars Prevention Curriculum 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Afterschool Program with All Stars Prevention Curriculum, Location 
offered in five Baltimore middle schools in 2005-06, aimed LOCALE REGION 
to increase school attendance and academic performance 
and reduce conduct problems. The program included 
traditional afterschool activities (snacks, sports, crafts), 
academic assistance, and the All Stars curriculum. The 


All Stars Curriculum focuses on reducing substance abuse 


and aggressive behavior and increasing social competency 


Northeast 
skills. The program was offered for 30 weeks, three days 
per week, three hours each day. 
Participants 
=] I en oA We ec Led CLASS GRADE TARGET STUDENTS 
Grades 6-8 General Population 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Attendance & Enrollment 
General Achievement 
School Engagement 


Social & Emotional Competencies 


, 58% Low-Income 71% Black 
NOT MEASURED FOR THIS INTERVENTION: Mathematics . 
Achievement, Other Achievement, Physical Activity/Health, Promotion 17% White 
& Graduation, Reading/ELA Achievement, and Science Achievement 12% Other 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Academic enrichment, arts/ SETTING: School site 


ion tan nd violen r ntion, 
recreation, substance use and violence preventio DELIVERY FORMAT: Group format 
social development 


STAFFING: Program staff 
BEST PRACTICES USED: Not reported 


DURATION: 30 weeks, 3 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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MULTICOMPONENT PROGRAMS | AFTERSCHOOL PROGRAM WITH ALL STARS PREVENTION CURRICULUM 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of an afterschool program for students in grades 6-8 at five Baltimore middle 
schools during the 2006-07 school year. Among students who signed up for the program, half were randomly assigned to participate 
and half were assigned to the control group. Outcomes for students randomized to the program were compared to outcomes for 
students randomized to the comparison group. The comparison students experienced “business as usual,” which could have included 


other school- and community-sponsored afterschool activities. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; , ces Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Days attended school > ) Grades 6-8 410 0.06 (0.10) 


Average effect size for Attendance & Enrollment outcomes at Tiers I-IIl 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . : Ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Academic performance 
composite (GPA, test > ) Grades 6-8 410 0.06 (0.10) 
scores, teacher rating) 
Average effect size for General Achievement outcomes at Tiers I-Ill 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


SCHOOL ENGAGEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : : pe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Evaluator-created 
measure of attachment > ) Grades 6-8 412 0.08 (0.10) 
to school 
Average effect size for School Engagement outcomes at Tiers I-Ill 0.08 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | AFTERSCHOOL PROGRAM WITH ALL STARS PREVENTION CURRICULUM 


SOCIAL AND EMOTIONAL COMPETENCES 


Evaluator-created 
measure of social 
competence (goal 0) 0) Grades 6-8 396 0.09 (0.10) 
setting, decision- 
making, impulsiveness) 


Average effect size for Social & Emotional Competencies outcomes at Tiers I-lll 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATIONS: Gottfredson, D. C., Cross, A. B., Wilson, D. M., Rorie, M., & Connell, N. (2010). Effects of participation in afterschool programs for middle 
school students: A randomized trial. Journal of Research on Educational Effectiveness, 3, 282-313. 


Cross, A. B., Gottfredson, D. C., Wilson, D. M., Rorie, M., & Connell, N. (2009). The impact of afterschool programs on the routine activities of middle 
school students: Results from a randomized, controlled trial. Criminology and Public Policy, 8, 391-412. 


Son, D. (2010). Racial differences in the effectiveness of after school programs (Unpublished master’s thesis). University of Maryland, College Park, MD. 


Cross, A. B. (2009). An experimental evaluation of after school program participation on problem behavior outcomes: Does pre-existing risk moderate 
the effects of program participation? (Unpublished doctoral dissertation). University of Maryland, College Park, MD. 


Gottfredson, D. C., Cross, A. B., Wilson, D. M., Rorie, M., & Connell, N. (2010). An experimental evaluation of the All Stars Prevention Curriculum in a 
community after school setting. Prevention Science, 11, 142-154. 
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MULTICOMPONENT PROGRAMS 


Santa Ana After School Learning and 


Safe Neighborhoods Partnerships Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Santa Ana After School Learning and Safe Neighborhoods Location 


Partnerships Program was implemented at four urban middle LOCALE REGION 
schools in the Santa Ana Unified School District in 1999-2000. 

The district partnered with community organizations to provide 
multicomponent afterschool programming that included home- 


work help and academic enrichment as well as recreational 
and other enrichment programming. The program was offered 
for the full school year, three hours each day. 


West 
Participants 
=] I en oA | ae We ec Led CLASS GRADE TARGET STUDENTS 
Grades 6-8 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Attendance & Enrollment 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- ec0¢ Lawinwwne 96% Latino 


ment, Other Achievement, Physical Activity/Health, Promotion and 
Graduation, School Engagement, Science Achievement, and Social & 
Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Sports, homework help SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Certified teachers, volunteers, program staff 
DURATION: Full school year, 3 hours/day 


IMPLEMENTATION SUPPORT: Provider training 
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MULTICOMPONENT PROGRAMS | SANTA ANA AFTER SCHOOL LEARNING AND SAFE NEIGHBORHOODS PARTNERSHIPS PROGRAM 


Study Details 


STUDY DESCRIPTION 

This study uses a quasi-experimental design to estimate the impact of attending the Santa Ana After School Learning and Safe 
Neighborhoods Partnerships Program on students at four middle schools in Santa Ana, California. Outcomes for participants were 
compared to outcomes for students who attended the same school but did not participate in the program. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. The 


study was conducted in 1999-2000. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; i ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Percentage of school 
co co Grades 6-8 1,358 -0.03 (0.06) 


days attended 


Average effect size for Attendance & Enrollment outcomes at Tiers I-IIl | 20s 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome ‘ ; See Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Stanford Achievement 
oo ro) Grades 6-8 1,358 -0.06 (0.06) 


Test 9 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl | 208 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 
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MULTICOMPONENT PROGRAMS | SANTA ANA AFTER SCHOOL LEARNING AND SAFE NEIGHBORHOODS PARTNERSHIPS PROGRAM 


READING/ELA ACHIEVEMENT 


Stanford Achievement 
Test 9 - reading oo oo Grades 6-8 1,358 -0.02 (0.06) 
comprehension 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Prenovost, J. K. E. (2001). A first-year evaluation of afterschool learning programs in four urban middle schools in the Santa Ana Unified 
School District (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3007008) 
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MULTICOMPONENT PROGRAMS 


School-to-Jobs 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The School-to-Jobs program, offered in one middle school in Location 
Detroit, Michigan from 1995-99, aimed to shift the self-per- 
ceptions and school involvement of low-income urban African 


LOCALE REGION 


American middle school students. The program consisted of 
small-group activities that were designed to create a positive 


peer-based experience in which African American racial 
identity was implicitly tied to academic achievement. The 


program was offered for nine weeks, one day per week, for 1.5 


West 

hours each day. 

Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 

Grade 8 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
School Engagement 

NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ene Mathematics penlevement, Other achievement ical Not Reported 100% Black 
Activity/Health, Promotion and Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, Science Achievement, and Social & 
Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: None reported SETTING: School site 
BEST PRACTICES USED: Sequenced, active learning, DELIVERY FORMAT: Group format 


iti It-st nt relationships, family engagement, wus : 
ena Saree enter ee STAFFING: Specialized staffing, volunteers 
respect for student cultures 


DURATION: 9 weeks, 1 day/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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MULTICOMPONENT PROGRAMS | SCHOOL-TO-JOBS 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the School-to-Jobs program, offered to eighth grade students in one middle school in 
Detroit. Outcomes for program participants are compared to outcomes for non-participants. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study was conducted 
in 1995-96, 1997-98, and 1998-99. 


SCHOOL ENGAGEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ane Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


School bonding 


(Cernkovich and oo co Grade 8 148 0.35 (0.18) 
Giordano, 1992) 


Average effect size for School Engagement outcomes at Tiers I-Ill 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Oyserman, D., Terry, K., & Bybee, D. (2002). A possible selves intervention to enhance school involvement. 
Journal of Adolescence, 25, 313-326. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES PAT PS 


MULTICOMPONENT PROGRAMS 


NIT iam Olivmelai(oitcn (1 Oly) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Support Our Students (SOS) program was a communi- Location 
ty-based afterschool initiative offered in North Carolina from LOCALE REGION 
1998-2001 that aimed to improve academic performance, 
reduce juvenile crime, and improve students’ attitudes and 
behaviors. SOS programs include at least four enrichment 
activities and one hour of homework time. Activities include 


dance clubs, drama clubs, intramural sports, or music activities. 


SOS is a multi-year program, but no other information about 


Southeast 
program duration is reported. 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grade 6 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


9 a 9 : 
ment, Other Achievement, Physical Activity/Health, Promotion & 96% Low-Income 81% White 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Homework help, arts, SETTING: Not reported 


t 
ane DELIVERY FORMAT: Not reported 


BEST PRACTICES USED: Not 
enrepenicd STAFFING: Not reported 
DURATION: Multi-year program 


IMPLEMENTATION SUPPORT: None reported 
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MULTICOMPONENT PROGRAMS | SUPPORT OUR STUDENTS (SOS) 


Study Details 


STUDY DESCRIPTION 

The quasi-experiment compared outcomes for low-achieving sixth-grade students who participated in North Carolina’s Support Our 
Students (SOS) afterschool program for three consecutive years (1998-99 through 2000-01), through eighth grade. The comparison 
students did not participate in SOS during those years. Outcomes for participants were compared to those for non-participants. 
The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; , ao Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


North Carolina 
End-of-Grade 8 8 Grade 8 410 -0.08 (0.10) 
Test - mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill -0.08 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome , ; eee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
North Carolina 
End-of-Grade Test - 8 8 Grade 8 410 -0.05 (0.10) 


reading 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Roukema, R. A. (2005). The impact of the Support Our Students (SOS) afterschool program on the achievement of middle-grade students at 
risk of academic failure (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3195186) 
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MULTI-STUDY PROGRAM SUMMARY 


21st Century Community Learning Centers 


(West Virginia — High School) 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


West Virginia’s 21st Century Community Learning No Effect 


Centers aimed to improve math and reading Overall, the studies found neither improved nor 


achievement of low-income students. Programs negative outcomes for 


served K-12 students throughout the state and were Mathematics Achievement 


operated by 33 separate programs. Afterschool 


programs offered a range of activities including Reading/ELA Achievement 


STEM programming, vocational enrichment, and 
professional presenters on a range of topics. A 
summer camp was also offered. 


This review of 21st Century Community Learning Centers (West Virginia — High School) identified 
three studies of this program (two manuscripts report the same study). The findings from each of these 
studies are presented individually on pages 276 — 284 of this evidence guide. 


« White, L., & Hixson N. (2014). 21st Century Community Learning Centers 2013: A quasi-experimental investigation 
of program impacts on student achievement in mathematics and reading/language arts. Charleston, WV: West 
Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


« White, L. J. (2014). 27st Century Community Learning Centers: A descriptive evaluation for 2012-13. Charleston, WV: 
West Virginia Department of Education, Division of Teaching and Learning, Office of Research. 


« White, L. J., Hammer, P. C., & Whisman, A. (2015). 21st Century Community Learning Centers, 2014-2015: A 
descriptive evaluation. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Assessment. 


« White, L. J., & Whisman, A. (2015). 21st Century Community Learning Centers 2014: A quasi-experimental 
investigation of program impacts on student achievement in mathematics and reading/language arts. 
Charleston, WV: West Virginia Department of Education, Division of Teaching and Learning, Office of Research. 
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21st Century Community Learning Centers 


(West Virginia - High School - 2012-13) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2012-13, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 9-11 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion and Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year, attendance range=2-93 days 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact for West Virginia students in grades 9-11 of participating in 21st Century Community Learning 
Center programs during 2012-13. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 


ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 9 168 0.06 (0.15) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 10 96 0.03 (0.20) 
Test 2 - mathematics 
West Virginia 
Educational Standads 8 8 Grade 11 96 0.16 (0.20) 
Test 2 - mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 0.08 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #1 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome F ‘ pet Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


West Virginia 
Educational Standards 
Test 2 - English 
language arts 


§ § Grade 9 168 0.15 (0.15) 


West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 10 96 0.04 (0.20) 


language arts 


West Virginia 
Educational Standards 
Test 2 - English 
language arts 


§ § Grade 11 96 0.36 (0.21) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: White, L., & Hixson N. (2014). 27st Century Community Learning Centers 2013: A quasi-experimental investigation of program impacts 
on student achievement in mathematics and reading/language arts. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Research. 


White, L. J. (2014). 21st Century Community Learning Centers: A descriptive evaluation for 2012-13. Charleston, WV: West Virginia Department of 
Education, Division of Teaching and Learning, Office of Research. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #2 


21st Century Community Learning Centers 
(West Virginia - High School - 2013-14) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2013-14, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 9-11 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year, attendance range=2-93 days 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 9-11 of participating in 21st Century Community Learning 
Center programs during 2013-14. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; Prael Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 9 134 0.26 (0.17) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 10 116 0.08 (0.19) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 11 90 0.24 (0.21) 


Test 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #2 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; y pet Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 
Test 2 - English g g Grade 9 134 0.03 (0.17) 
language arts 
West Virginia 
Educational Standards 
Test 2 - English 8 8 Grade 10 116 0.14 (0.19) 
language arts 
West Virginia 
Educational Standards 
§ g Grade 11 90 0.19 (0.21) 


Test 2 - English 
language arts 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: White, L. J., & Whisman, A. (2015). 21st Century Community Learning Centers 2014: A quasi-experimental investigation of program impacts 
on student achievement in mathematics and reading/language arts. Charleston, WV: West Virginia Department of Education, Division of Teaching and 
Learning, Office of Research. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #3 


PAR a Oo iigaGunviiinting Me lentitm olan 


(West Virginia - High School - 2014-15) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


West Virginia’s 21st Century Community Learning Centers, Location 


offered in 2014-15, aimed to improve math and reading LOCALE REGION 
achievement of low-income students. Programs served K-12 
students throughout the state and were operated by 33 
separate programs. Afterschool programs offered a range of 


activities including STEM programming, vocational enrichment, 


and professional presenters on a range of topics. The pro- 


grams were offered for the full school year. 


Suburban, Rural, Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 9-11 Low-Income 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Mathematics Achievement 
Reading/ELA Achievement 
NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Other Achievement, Physical Activity/Health, Promotion & Not Reported Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Multiple programs SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group and one-to-one formats 


STAFFING: Program staff, volunteers 
DURATION: Full school year 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #3 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact for West Virginia students in grades 9-11 of participating in 21st Century Community Learning 
Center programs during 2014-15. Outcomes for students who participated in the programs were compared to outcomes for non-partic- 
ipants. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
West Virginia 
Educational Standards 8 8 Grade 9 66 0.01 (0.17) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 10 98 0.11 (0.19) 
Test 2 - mathematics 
West Virginia 
Educational Standards 8 8 Grade 11 72 -0.18 (0.21) 


Test 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY 21ST CENTURY COMMUNITY LEARNING CENTERS (WEST VIRGINIA — HIGH SCHOOL) | SUMMARY #3 


READING/ELA ACHIEVEMENT 


West Virginia 
Educational Standards 
Test 2 - English 
language arts 


8 8 Grade 9 33 0.10 (0.17) 


West Virginia 
Educational Standards 
Test 2 - English 
language arts 


8 8 Grade 10 98 0.34 (0.19) 


West Virginia 
Educational Standards 
Test 2 - English 
language arts 


8 8 Grade 11 72 0.11 (0.21) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: White, L. J., Hammer, P. C., & Whisman, A. (2015). 21st Century Community Learning Centers, 2014-2015: A descriptive evaluation. Charles- 
ton, WV: West Virginia Department of Education, Division of Teaching and Learning, Office of Assessment. 
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MULTICOMPONENT PROGRAMS 


Baltimore Community Schools (High School) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Baltimore Community Schools programs, offered in 2012-14, Location 

took place at community schools serving low-income students LOCALE REGION 
in grades K-12. The programs were intended to promote 
academic success, health, social and emotional development, 
high school graduation, and college and career readiness. 
Programs were provided by various types of organizations and 
offer academic support, recreation, arts and sports activities. 


The programs were offered for the full school year, up to five 


Northeast 
days per week, for up to 4.5 hours per day. 
Participants 
= I 4 en BAVA Se Wed eV ed = CLASS GRADE TARGET STUDENTS 
Grades 9-12 Low-Income 
No Effect 
The study found neither improved nor negative outcomes for SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Attendance & Enrollment 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical 
Activity/Health, Promotion & Graduation, Reading/ELA Achievement, 
School Engagement, Science Achievement, and Social & Emotional 6% Latino 
Competencies 


95% Low-Income 85% Black 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Sports, homework help SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Full school year, up to 5 days/week, 3-4.5 hours/day 


IMPLEMENTATION SUPPORT: Ongoing support 
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MULTICOMPONENT PROGRAMS | BALTIMORE COMMUNITY SCHOOLS (HIGH SCHOOL) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool programs in community schools, administered by the Family League of 
Baltimore. Outcomes for students who attended afterschool programs for at least 200 hours were compared to outcomes for students 
who did not attend the programs and did not attend a community school. The comparison students experienced “business as usual,” 
which could have included other school- and community-sponsored afterschool activities. The study was conducted in 2012-14. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome z ; ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Chronic absenteeism 
(more than 20 8 8 Grades 9-12 324 0.17 (0.11) 


days absent) 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 0.17 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Olson, L. S. (2014). A first look at community schools in Baltimore. Baltimore, MD: Baltimore Education Research Consortium. 


Olson, L. S., Connolly, F., & Kommajesula, A. H. (2013). Family League 2011-12 out of school time programs in Baltimore city. Baltimore, MD: Baltimore 
Education Research Consortium. 
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Physical Activity and Health 


PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


America SCORES 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


America SCORES, offered in three schools in a large urban Location 
school district in 2009-10, was a soccer program that aimed to LOCALE REGION 
build competencies and skills, including teamwork, leadership, 

and academic commitment. In addition to soccer, the program 

also offered creative writing and service learning projects. 

Students spent two to three days per week in soccer drills or Hy | 

games for up to two hours each day, and on the other days 


engaged in creative writing (fall) and service learning (spring). 


Urban West 
The program was offered for 24 weeks (12 weeks in the fall 
and 12 weeks in the spring), five days per week. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 4-5 General Population 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


0 : 
Promotion & Graduation, Reading/ELA Achievement, met Reported AoE aang 
Attendance & Enrollment, School Engagement, Science 32% Asian 
Achievement, and Social & Emotional Competencies. 12% Black 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Sports, tutoring SETTING: School site 

BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 12 weeks in Fall and Spring, 5 days/week 


IMPLEMENTATION SUPPORT: Program manual, 
provider training, ongoing support 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | AMERICA SCORES 


Study Details 


STUDY DESCRIPTION 

In this study, three schools were randomized to participate in America SCORES and three schools were randomized to the comparison 
group. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. The study was conducted in 2009-10. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Moderate-to-vigorous : , 
: 2 Tier Il Tier Ill Grades 4-5 150 0.57 (0.17)* 
physical activity 
Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 0.57 


* Statistically significant at p<.05 


CITATION: Madsen, K., Thompson, H., Adkins, A., & Crawford, Y. (2013). School-community partnerships: A cluster-randomized trial of an afterschool 
soccer program. JAMA Pediatrics, 167(4), 321-326. 
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MULTI-STUDY PROGRAM SUMMARY 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


Positive Effect 


The FITKids (Fitness Improves Thinking) program 
consisted of physical activity including rotating 
among stations and playing games that developed 
particular physical skills, such as dribbling. The 
program also included a healthy snack and a brief 
education component focused on nutrition. 


NOTE: The CATCH Kids Club program was a 
precursor to the FITKIDs program and is 
substantially similar. Therefore, findings from 
CATCH studies are included in the overall 
effectiveness summary for this program. 


‘This review of FIT Kids identified ten studies of this pro- 
gram. The findings from each of these studies are presented 
individually on pages 291 — 304 of this evidence guide. 


de Heer, H. (2009). Project L.E.A.N.: An afterschool health 
and exercise program for elementary school children in El 
Paso, Texas (Doctoral dissertation). Retrieved from ProQuest 
Dissertations and Theses Databse (UMI No. 3358902) 


de Heer, H. (2009). A path analysis of an afterschool health 
and exercise program for elementary school children in 

El Paso, Texas (Master’s thesis). Retrieved from ProQuest 
Dissertations and Theses Database (UMI No. 1462432) 


de Heer, H. D., Koehly, L., Pederson, R., & Morera, O. 
(2011). Effectiveness and spillover of an afterschool health 
promotion program for Latino elementary school children. 
American Journal of Public Health, 101, 1907-1913. 


Gutin, B., Yin, Z., Johnson, M., & Barbeau, P. (2008). Prelim- 
inary findings of the effect of a 3-year afterschool physical 
activity intervention on fitness and body fat: The Medical 
College of Georgia FitKid Project. International Journal of 
Pediatric Obesity, 3, 3-9. 


Hillman, C. H., Pontifex, M. B., Castelli, D. M., Khan, N. A., 
Raine, L. B., Scudder, M. R., .. ., Kamijo, K. (2014). Effects of 
the FITKids randomized controlled trial on executive control 
and brain function. Pediatrics, 134, e1063-e1071. 


Overall, the studies found at least one improved 


outcome with no overriding contrary evidence for 


Physical Activity/Health 
No Effect 


Overall, the studies found neither improved nor 
negative outcomes for 

Mathematics Achievement 

Other Achievement 

Reading/ELA Achievement 


Science Achievement 


Monti, J. M., Hillman, C. H., & Cohen, N. J. (2012). Aerobic 
fitness enhances relational memory in preadolescent 
children: The FITKids randomized control trial. Hippocampus, 
22(9), 1876-1882. 


Rumph, M. K. (2011). Evaluation of CATCH Kids Club after 
school program: A nutrition and physical fitness intervention 
for third, fourth, and fifth graders (Master’s thesis). Retrieved 
from ProQuest Dissertations and Theses Database (UMI No. 
1500125) 


Yin, Z., Hanes, J., Moore, J. B., Humbles, P., Barbeau, P., & 
Gutin, B. (2005). An afterschool physical activity program 
for obesity prevention in children: The Medical College 
of Georgia FitKid Project. Evaluation and the Health 
Professions, 28, 67-89. 


Yin, Z., Moore, J. B.., Johnson, M. H., Barbeau, P., Cavnar, 

M., Thornburg, J., & Gutin, B. (2005). The Medical College 

of Georgia FitKid Project: The relations between program 
attendance and changes in outcomes in year 1. International 
Journal of Obesity, 29, S40-S45. 


Yin, Z., Gutin, B., Johnson, M. H., Hanes, J., Moore, J. B., 
Cavnar, M.,..., Barbeau, P. (2005). An environmental 
approach to obesity prevention in children: Medical College 
of Georgia FitKid Project year 1 results. Obesity Research, 13, 
2153-2160. 


MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #1 


CATCH Kids Club program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The CATCH (Coordinated Approach to Children’s Health) Kids Location 
Club program, offered in 13 elementary schools in Oklahoma 


LOCALE REGION 
during 2009-10, was a precursor to the FITKids program. 


CATCH Kids Club focused on nutrition and physical activity for 
low-income students. The program aimed to improve children’s 
physical activity levels as well as their knowledge, attitudes, 


and behaviors regarding healthy eating. The program was 
offered for 16 weeks, five days per week. 


Not Reported Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 

Physical Activity/Health 
NOT MEASURED FOR THIS INTERVENTION: General 


Achievement, Mathematics Achievement, Other Achievement, 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


0, 9 
Promotion & Graduation, Reading/ELA Achievement, Nor Peported oui elute Svuaeetan 
Attendance & Enrollment, School Engagement, Science 24% Black 11% Other 
Achievement, and Social & Emotional Competencies 12% Latino 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Sports, nutrition education SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 


STAFFING: Volunteers 
DURATION: 16 weeks, 5 days/week 


IMPLEMENTATION SUPPORT: Program manual 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the CATCH Kids Club for students in grades 3-5 in 13 elementary schools in Lawton, 
Oklahoma. Outcomes for program participants are compared to outcomes for students attending other schools in the district that did 
not participate in the program. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. The study was conducted in 2009-10. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome : ; en Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Progressive Aerobic 
Cardiovascular 
Endurance Run (PACER . . 
( ) Tier Ill Tier Ill Grades 3-5 112 0.41 (0.19) * 


within or above 
Fitnessgram Healthy 
Fitness Zone 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


CITATION: Rumph, M. K. (2011). Evaluation of CATCH Kids Club after school program: A nutrition and physical fitness intervention for third, fourth, and 
fifth graders (Master’s thesis). Retrieved from ProQuest Dissertations and Theses Databse (UMI No. 1500125) 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #2 


CATCH /Bienestar (El Paso — A) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The CATCH/Bienestar afterschool program, offered in six Location 


elementary schools in El Paso, Texas during 2008, focused LOCALE REGION 
on health and physical activity and was targeted to Latino 

elementary school students in grades 3-5. The program used 

two interventions: Bienestar, a bilingual health education 

curriculum that was culturally tailored to Mexican Americans, i | 

and an adaptation of the CATCH (Coordinated Approach to 


Children’s Health) curriculum, a precursor to FITKids, which 


Urban Southwest 

focuses on physical activity and games. The program was 
offered for 12 weeks, two days per week, for up to 1.5 hours 
each day. 

Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 3-5 Latino Students 

SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


0, = 0 ‘ 0, 
Promotion & Graduation, Reading/ELA Achievement, 98% Low-Income 27% Latino 3% Black 
Attendance & Enrollment, School Engagement, Science 12% White 6% Other 
Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: Sports SETTING: School site 

BEST PRACTICES USED: Sequenced, respect for DELIVERY FORMAT: Group format 


student cultures 
STAFFING: Certified teachers, program staff 


DURATION: 12 weeks, 2 days/week, 1-1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, staff-student cultural similarity 
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MULTI-STUDY PROGRAM SUMMARY Ge ON 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial was conducted in six elementary schools in El Paso, Texas, with third, fourth, and fifth grade classrooms 
randomized to the program or to the comparison condition. The comparison students experienced “business as usual,” which could 
have included other school- and community-sponsored afterschool activities. The study was conducted in 2008. 


*NOTE: This study used a different sample of students from the CATCH/Bienestar (El Paso-B) study. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; 5 ne Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index 8 8 Grades 3-5 931 -0.01 (0.07) 


Progressive Aerobic 
Cardiovascular 8 8 Grades 3-5 931 0.06 (0.07) 
Endurance Run 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: de Heer, H. (2009). Project L.E.A.N.: An afterschool health and exercise program for elementary school children in El Paso, Texas (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3358902) 


de Heer, H. (2009). A path analysis of an afterschool health and exercise program for elementary school children in El Paso, Texas (Master’s thesis). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 1462432) 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #3 


CATCH /Bienestar (El Paso - B) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The CATCH/Bienestar afterschool program, offered in six Location 


elementary schools in El Paso, Texas in 2008, focused on LOCALE REGION 
health and physical activity and was targeted to Latino 

elementary school children (3rd-5th grade). The program 

used two interventions: Bienestar, a bilingual health education 

curriculum that was culturally tailored to Mexican Americans, i | 

and an adaptation of the CATCH (Coordinated Approach to 


Children’s Health) curriculum, a precursor to FITKids, which 


Urban Southwest 
focuses on physical activity and games. 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 3-5 Latino Students 
SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 
Physical Activity/Health 
NOT MEASURED FOR THIS INTERVENTION: General 


Achievement, Mathematics Achievement, Other Achievement, 


0, = 0, 
Promotion & Graduation, Reading/ELA Achievement, 72% Low-Income 100% Latino 
Attendance & Enrollment, School Engagement, Science 

Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: Sports SETTING: School site 

BEST PRACTICES USED: Sequenced, respect for DELIVERY FORMAT: Group format 


student cultures 
STAFFING: Certified teachers, program staff 


DURATION: 12 weeks, 2 days/week, 1-1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, staff-student cultural similarity 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #3 


Study Details 


STUDY DESCRIPTION 
This randomized controlled trial estimates the impact of a health education and physical fitness program for students in six elementary 


schools in El Paso, Texas. Comparison students attended an afterschool program but did not participate in this intervention. The study 


was conducted in 2008. 


*NOTE: This study used a different sample of students from the CATCH/Bienestar (El Paso-A) study. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 


Number of Effect size 
Outcome ; ; ne Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index co co Grades 3-5 568 -0.01 (0.08) 


Progressive Aerobic 
Cardiovascular > ) Grades 3-5 568 0.03 (0.08) 


Endurance Run 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: de Heer, H. D., Koehly, L., Pederson, R., & Morera, O. (2011). Effectiveness and spillover of an afterschool health promotion program for Latino 
elementary school children. American Journal of Public Health, 101, 1907-1913. 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #4 


FIT Kids (Southeast) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This FITKids afterschool program, offered at 18 schools in Location 
Georgia, aimed to increase fitness and reduce obesity of chil- LOCALE REGION 
dren in grades 3-5. The program offered moderate-to-vigorous 
physical activity for extended periods of time (40-80 minutes) i 
after school. Activities included rotating among stations and 
playing games that developed specific physical skills. The (=) B 
program also included a healthy snack and a brief education 
component focused on nutrition. The program was offered for Saharan) Rural’ Unban Sactheast 
the full school year, five days per week, two hours each day. 
The implementation year was not reported. 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grade 3 General Population 


Positive Effect 


The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 
Physical Activity/Health 


No Effect 


The study found neither improved nor negative outcomes for 
Mathematics Achievement 


Other Achievement 


Reading/ELA Achievement 


: F 68% Low-Income 61% Black 31% White 
Science Achievement 
2% Latino 5% Other 
NOT MEASURED FOR THIS INTERVENTION: General . 
2% Asian 


Achievement, Other Achievement, Promotion & Graduation, School 
Engagement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 


MAIN COMPONENTS: Homework help, organized team SETTING: School site 


Hyalesi asset 
sports, unstructured physical activities DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Not reported STAFFING: Certified teachers 
DURATION: Full school year, 5 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #4 


Study Details 


STUDY DESCRIPTION 

In this randomized controlled trial, 18 schools in Georgia were randomly assigned to receive the intervention or to serve as comparison 
schools. The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. The implementation year was not reported. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; . ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced 
g g Grade 3 426 -0.11 (0.10) 


Competency Test - 
mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


OTHER ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; y ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced 
g g Grade 3 426 -0.01 (0.10) 


Competency Test - 
social studies 


Average effect size for Other Achievement outcomes at Tiers I-IIl 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #4 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome E . home Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
B 
ees 00 00 Grade 3 447 0.03 (010) 
mass index 
Bone 
SiRcHE Sno: 00 © Grade 3 447 0.13 (0.10) 
Biestolte oo oo Grade 3 447 0.19 (0.10) 
blood pressure 
HDL cholesterol co co Grade 3 447 0.15 (0.10) 
Heart rate Tier Il Tier Il Grade 3 447 0.27 (0.10) * 
Percent body fat co co Grade 3 447 0.08 (0.10) 
Systolic : ‘ 
Tier Il Tier Il Grade 3 447 0.19 (0.10) * 
blood pressure 
Waist 
a oo oo Grade 3 447 0.04 (010) 


circumference (cm) 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill pom 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #4 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome Group 


Without site and With site and students (Standard error) 
sample criteria sample criteria 


Georgia Criterion- 
Referenced 
Competency Test - 
English language arts 


§ § Grade 3 426 0.00 (0.10) 


Georgia Criterion- 
Referenced 
Competency Test - 
reading 


§ § Grade 3 426 0.01 (0.10) 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome a i an Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Georgia Criterion- 
Referenced 
g § Grade 3 426 -0.07 (0.10) 
Competency Test - 
science 
Average effect size for Science Achievement outcomes at Tiers I-lll -0.07 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Gutin, B., Yin, Z., Johnson, M., & Barbeau, P. (2008). Preliminary findings of the effect of a 3-year afterschool physical activity intervention on 
fitness and body fat: The Medical College of Georgia FITKid Project. International Journal of Pediatric Obesity, 3, 3-9. 


Yin, Z., Hanes, J., Moore, J. B., Humbles, P., Barbeau, P., & Gutin, B. (2005). An afterschool physical activity program for obesity prevention in children: 
The Medical College of Georgia FITKid Project. Evaluation and the Health Professions, 28, 67-89. 


Yin, Z., Moore, J. B.., Johnson, M. H., Barbeau, P., Cavnar, M., Thornburg, J., & Gutin, B. (2005). The Medical College of Georgia FITKid Project: The 
relations between program attendance and changes in outcomes in year 1. International Journal of Obesity, 29, S40-S45. 


Yin, Z., Gutin, B., Johnson, M. H., Hanes, J., Moore, J. B., Cavnar, M.,..., Barbeau, P. (2005). An environmental approach to obesity prevention in children: 
Medical College of Georgia FITKid Project year 1 results. Obesity Research, 13, 2153-2160. 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #5 


FIT Kids (Midwest - A ) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This FITKids afterschool program, offered at a recreational Location 


facility at the University of Illinois campus from 2009-2013, LOCALE REGION 
aimed to increase fitness and reduce obesity of children 

in grades 3-5. The program offered moderate-to-vigorous 

physical activity for extended periods of time (average 70 

minutes), followed by a healthy snack and a brief education i | 

component focused on nutrition and low-organization games 


that developed specific physical skills. The program was 


Urban Midwest 

offered for the full school year, five days per week, two hours 
each day (a total of 150 days). 

Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Ages 7-9 General Population 

SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


; i : ; 43% Low-Income 55% White 
Promotion & Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, School Engagement, Science 27% Black 
Achievement, and Social & Emotional Competencies 15% Asian 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Sports, social skills development, SETTING: Community site 


neue dueaten DELIVERY FORMAT: Group format 


BEST PRACTICES SED: Nontepened STAFFING: Certified teachers, volunteers 
DURATION: Full school year, 5 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, ongoing support 
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MULTI-STUDY PROGRAM SUMMARY Ge ON Me) 


Study Details 


STUDY DESCRIPTION 


This randomized controlled trial estimates the impact of FITKids for students ages 7-9. Outcomes for participants are compared to 


those for non-participants. The comparison students experienced “business as usual,” which could have included other school- and 


community-sponsored afterschool activities. The study was conducted from 2009-13. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Outcome ; : egy Group 
Without site and With site and 
sample criteria sample criteria 
Body mass index > ) Ages 7-9 
Maximal oxygen 
consumption, normed ) ) Ages 7-9 


percentile 


Number of Effect size 
students (Standard error) 
221 0.15 (0.14) 
221 0.21 (0.14) 


Average effect size for Physical Activity/Health outcomes at Tiers I-III 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Hillman, C. H., Pontifex, M. B., Castelli, D. M., Khan, N. A., Raine, L. B., Scudder, M. R.,..., Kamijo, K. (2014). Effects of the FITKids randomized 


controlled trial on executive control and brain function. Pediatrics, 134, e1063-e1071. 
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MULTI-STUDY PROGRAM SUMMARY FITKIDS | SUMMARY #6 


FIT Kids (Midwest - B ) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This FITKids afterschool program, offered in Illinois, aimed to Location 


increase fitness and reduce obesity of children in grades 3-5. LOCALE REGION 
The program offered moderate-to-vigorous physical activity for 
extended periods of time (average 70 minutes), followed by 

a healthy snack and a brief education component focused on 
nutrition and low-organization games that developed specific 
physical skills. The program was offered for the full school 


year, five days per week, two hours each day. The program 


Not Reported 
intervention year is not reported. 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


Not R 
Promotion & Graduation, Reading/ELA Achievement, Not Reported a eported 
Attendance & Enrollment, School Engagement, Science 

Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Sports, social skills development, SETTING: School and community sites 


neuen dueaten DELIVERY FORMAT: Group format 


BEST PRACTICES USED: Not reported STAFFING: Volunteers 
DURATION: Full school year, 5 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Program manual 
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MULTI-STUDY PROGRAM SUMMARY eNO ea) 


Study Details 


STUDY DESCRIPTION 

This study randomly assigned students to participate in the program or to the comparison group. Outcomes for participants are 
compared to those for non-participants. The comparison students experienced “business as usual,” which could have included other 
school- and community-sponsored afterschool activities. The implementation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; : heyy Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Maximal oxygen 
ue § § Age 9 44 0.54 (0.31) 


consumption 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Monti, J. M., Hillman, C. H., & Cohen, N. J. (2012). Aerobic fitness enhances relational memory in preadolescent children: The FITKids 
randomized control trial. Hippocampus, 22(9), 1876-1882. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Georgia Prevention Institute Physical Activity Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Georgia Prevention Institute’s Physical Activity Program, Location 


offered in eight schools in Georgia, aimed to improve cardio- LOCALE REGION 


vascular fitness in Black elementary school girls. The program 


involved daily physical activity including skills instruction, 
aerobic activity, and strengthening and stretching activities. 
The program also included homework help. The program was 
offered for the full school year, five days per week, two hours 
each day. The implementation year is not reported. 


Not Reported Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


: SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Mixed Effects 


The study found a mix of improved outcomes and null or 
negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


0, 
Promotion & Graduation, Reading/ELA Achievement, Not Reported 100% Black 
Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Sports, homework help SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: Full school year, 5 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, ongoing support 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | GEORGIA PREVENTION INSTITUTE PHYSICAL ACTIVITY PROGRAM 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of an afterschool physical activity program for Black girls in grades 3-5 from eight 
schools in Georgia. Outcomes for students randomly assigned to participate in the program are compared to outcomes for students 
randomly assigned to the comparison group. The comparison students experienced “business as usual,” which could have included 
other school- and community-sponsored afterschool activities. The implementation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome z 4 en Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index > ) Grades 3-5 201 0.08 (0.14) 
Bone mineral density ) ) Grades 3-5 201 0.26 (0.14) 
Graded treadmill test Tier | Tier Ill Grades 3-5 201 0.31 (0.14)* 
Percentage body fat ) ) Grades 3-5 201 0.17 (0.14) 
Subcutaneous 
b inal adi 
cata eal nae ) ) Grades 3-5 110 0.10 (0.19) 
tissue (cubic 
centimeters) 
Vi | adi ti 
pede aes ) ) Grades 3-5 10 0.23 (0.20) 
(cubic centimeters) 
Waist 
ae 0 0 Grades 3-5 201 0.11 (0.14) 


circumference (cm) 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill | ow 


* Statistically significant at p<.05 


CITATION: Barbeau, P., Johnson, M. H., Howe, C. A., Allison, J., Davis, C. L., Gutin, B., & Lemmon, C. R. (2007). Ten months of exercise improves 
general and visceral adiposity, bone, and fitness in black girls. Obesity, 15, 2077-2085. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


@itacerimeitmertiits 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Girls in the Game program, offered at five schools in Location 
Chicago, aimed to foster social-emotional development and LOCALE REGION 
reduce BMI for urban, low-income Black and Latina elementary — 
school girls. The program sought to empower girls to make é i 
healthy choices through physical activity in traditional and ‘ i 
non-traditional sports, fitness activities, education activities | | jl : ~ ae = 
focused on health and nutrition, and leadership and life skills. pn " 
Materials were sent home for families to reinforce the program- Urban Midwest 
ming messages. The program was offered for the full school 
year, one day per week, 1.5 hours each day. 

Participants 

CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE ' Grades 3-5 ) ee ee) 


: SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
Mixed Effects 


The study found a mix of improved outcomes and null or 
negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


: : . é Not Reported 60% Latina 
Promotion & Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, School Engagement, Science 36% Black 
Achievement, and Social & Emotional Competencies 4% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Sports SETTING: School site 
BEST PRACTICES USED: Sequenced, active learning, DELIVERY FORMAT: Group format 


ene ngegeient STAFFING: Program staff 


DURATION: Full school year, 1 day/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, staff-student cultural similarity 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | GIRLS IN THE GAME 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of Girls in the Game for students in grades 3-5 at five schools in Chicago. Out- 
comes for those randomized to the program are compared to those who were randomized to the comparison group. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 


The implementation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 


Number of Effect size 
Outcome i 4 Hae Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 
Body mass index 8 8 Grades 3-5 68 0.06 (0.27 ) 

Consumption of fruits 
‘ 00 00 Grades 3-5 76 -0.02 (0.25) 

and vegetables 
Sueno a oo co Grades 3-5 76 0.36 (0.25 
unhealthy foods re GN 25) 
Physical activity Tier Il Tier Il Grades 3-5 76 0.54 (0.25)* 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 


* Statistically significant at p<.05 
co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Bohnert, A. M., & Ward, A. K. (2013). Making a difference: Evaluating the Girls in the Game (GIG) afterschool program. Journal of Early 
Adolescence, 33(1), 104-130. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


LA Sprouts 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


LA Sprouts, offered in Los Angeles in 2010, was a 12-week Location 

gardening, nutrition, and cooking intervention that encouraged LOCALE REGION 
healthy eating and aimed to reduce the risk of obesity and 

other metabolic disorders among Latino students in grades 

4-5. Nutrition lessons focused on increasing fruit and vegetable 

intake, including culturally relevant foods. Students prepared a Hy | 

sample recipe each week and participated in planting, grow- 

ing, maintaining, and harvesting organic fruits and vegetables. 


Urban West 
The program incorporated parents into several gardening and 
nutrition sessions. The program was offered for 12 weeks, one 
day per week, for 1.5 hours. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 4-5 Latino Students 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 
Promotion & Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 


94% of students at school 94% Latino 
qualify for free or re- 3% Asian 
duced-price lunch 3% White 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Nutrition education SETTING: Community site 
BEST PRACTICES USED: Sequenced, active learning, DELIVERY FORMAT: Group format 


famil ment, cultural elements 
Se eae een ! STAFFING: Program staff, specialized staffing 


DURATION: 12 weeks, 1 day/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | LA SPROUTS 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact of LA Sprouts for students in grades 4-5 in one elementary school in Los Angeles. Out- 
comes for participants are compared to those for non-participants. The comparison students experienced “business as usual,” which 


could have included other school- and community-sponsored afterschool activities. The study was conducted in 2010. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; . ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Added sugar (tsp) 00 co Grades 4-5 98 0.14 (0.21) 
Body mass index 8 8 Grades 4-5 104 0.04 (0.21) 
Caloric intake oo co Grades 4-5 98 0.06 (0.21) 
Dairy servings/day 00 co Grades 4-5 98 0.00 (0.21) 
Diastolic blood pressure co oo Grades 4-5 104 0.17 (0.21) 
Fiber intake Tier Il Tier Ill Grades 4-5 98 0.99 (0.23) * 
Fruit servings/day oo oo Grades 4-5 98 -0.25 (0.21) 
Meat servings/day oo oo Grades 4-5 98 0.06 (0.21) 
Bolen ca euyciel oo oo Grades 4-5 98 -0.11 (0.21) 
calories 
Percent fat calories oo oo Grades 4-5 98 0.19 (0.21) 
Percent protein calories oo oo Grades 4-5 98 0.19 (0.21) 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | LA SPROUTS 


PHYSICAL ACTIVITY/HEALTH CONTINUED 


Vegetable servings/day Tier Il Tier Ill Grades 4-5 98 0.60 (0.22) * 


Waist circumference 


co co Grades 4-5 104 0.04 (0.21) 

(cm) 
Weight (kg) 8 8 Grades 4-5 104 0.03 (0.21) 
Whole grains oz/day oo oo Grades 4-5 98 0.31 (0.21) 


Average effect size for Physical Activity/Health outcomes at Tiers I-IIl 


* Statistically significant at p<.05 
co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATIONS: Gatto, N. M., Ventura, E. E., Cook, L. T., Gyllenhammer, L. E., & Davis, J. N. (2012). LA Sprouts: A garden-based nutrition intervention 


pilot program influences motivation and preferences for fruits and vegetables in Latino youth. Journal of the Academy of Nutrition and Dietetics, 112, 
913-920. 


Davis, J. N., Ventura, E. E., Cook, L. T., Gyllenhammer, L. E., & Gatto, L. M. (2011). LA Sprouts: A gardening, nutrition, and cooking intervention for Latino 
youth improves diet and reduces obesity. Journal of the American Dietetic Association, 111, 1224-1230. 
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MULTI-STUDY PROGRAM SUMMARY 


Youth Fit for Life 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


This Youth Fit for Life program aimed to increase Positive Effect 
the health and fitness, mood, and self-appraisals of Overall, the studies found at least one improved 


students ages 9-12. The program involved students outcome with no overriding contrary evidence for 


in vigorous physical activity through non-compet- 
itive games and resistance training. Participants Physical Activity/Health 
also received health and nutrition education and 


self-management/self-regulatory skills training. 


This review of Youth Fit for Life identified 10 studies of this program. The findings from each of these 
studies are presented individually on pages 313 — 316 of this document. 
« Annesi, J. J. (2006). Relations of physical self-concept and self-efficacy with frequency of voluntary physical activity 


in preadolescents: Implications for afterschool care programming. Journal of Psychosomatic Research, 61(4), 
515-520. 


« Annesi, J. J., Faigenbaum A. D., Westcott, W. L. & Smith A. E. (2008). Relations of self-appraisal and mood changes 
with voluntary physical activity changes in African American preadolescents in an afterschool care intervention. 
Journal of Sports Science and Medicine, 7, 260-268. 


MULTI-STUDY PROGRAM SUMMARY YOUTH FIT FOR LIFE | SUMMARY #1 


Youth Fit for Life - YMCA 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This Youth Fit for Life program, offered at a YMCA in 2003, Location 
aimed to increase the health and fitness, mood, and self-ap- LOCALE REGION 


praisals of students ages 9-12. The program involved students 
in vigorous physical activity through non-competitive games 
and resistance training. Participants also received health and 
nutrition education and self-management/self-regulatory skills 
training. The program was offered for 12 weeks, three days per 


week, 45 minutes each day. Southeast 


Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


No Effect 


The study found neither improved nor negative outcomes for 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


‘ : ‘ ; Not Reported 70% Black 
Promotion & Graduation, Reading/ELA Achievement, . 
Attendance & Enrollment, School Engagement, Science 19% White 
Achievement, and Social & Emotional Competencies 11% Other 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Sports, social skills activities SETTING: Community site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 


STAFFING: Specialized staffing 
DURATION: 12 weeks, 3 days/week, 45 minutes/day 


IMPLEMENTATION SUPPORT: Program manual, provider 
training, ongoing support 
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MULTI-STUDY PROGRAM SUMMARY YOUTH FIT FOR LIFE | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of Youth Fit for Life (YFL) for a group of students attending a YMCA afterschool program. 
Outcomes for participants are compared to those for non-participants. The comparison students experienced “business as usual,” which 
could have included other school- and community-sponsored afterschool activities. The study was conducted in 2003. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome R i mae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Physical activity co oo Ages 9-12 81 0.36 (0.22) 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Annesi, J. J. (2006). Relations of physical self-concept and self-efficacy with frequency of voluntary physical activity in preadolescents: 
Implications for afterschool care programming. Journal of Psychosomatic Research, 61(4), 515-520. 
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MULTI-STUDY PROGRAM SUMMARY YOUTH FIT FOR LIFE | SUMMARY #2 


Youth Fit for Life — School Site 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This Youth Fit for Life program, offered at a school site, aimed Location 


to increase the health and fitness, mood, and self-appraisals of LOCALE REGION 
students ages 9-12. The program involved students in vigorous 
physical activity through non-competitive games and resis- 
tance training. Participants also received health and nutrition 
education and self-management/self-regulatory skills training. 


The program was offered for 12 weeks, three days per week, 


45 minutes each day. 


Not Reported Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


Positive Effect 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


0, 
Promotion & Graduation, Reading/ELA Achievement, Not Reported 100% Black 
Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Sports, social skills activities SETTING: School site 
BEST PRACTICES USED: Sequenced DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 12 weeks, 3 days/week, 45 minutes/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 315 


MULTI-STUDY PROGRAM SUMMARY YOUTH FIT FOR LIFE | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of Youth Fit for Life for Black preadolescents (average age 10 years). Outcomes for partici- 
pants are compared to those for non-participants. The comparison students experienced “business as usual,” which could have included 
other school- and community-sponsored afterschool activities. The implementation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome : : aera Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Physical activity Tier Il Tier Ill Ages 9-12 269 0.40 (0.12)* 


Average effect size for Physical Activity/Health outcomes at Tiers I-III 


* Statistically significant at p<.05 


CITATION: Annesi, J. J., Faigenbaum A. D., Westcott, W. L. & Smith A. E. (2008). Relations of self-appraisal and mood changes with voluntary physical 
activity changes in African American preadolescents in an afterschool care intervention. Journal of Sports Science and Medicine, 7, 260-268. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Fitness-Focused Afterschool Programs (California) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


These fitness-focused afterschool programs, offered by a Location 

community in the San Francisco Bay area from 2006-09, were LOCALE REGION 

offered by city departments, school districts, and four nonprofit 

organizations. Examples of fitness activities offered by these 

programs included soccer, dance, and yoga. More detailed 

information about the programs or their duration is 

not reported. 
Not Reported West 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 5-7 General Population 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 
Promotion & Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, School Engagement, Science 27% White 
Achievement, and Social & Emotional Competencies 


62% Low-Income 63% Latino 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Sports SETTING: School and community sites 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | FITNESS-FOCUSED AFTERSCHOOL PROGRAMS (CALIFORNIA) 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment uses administrative data from local afterschool providers and from local afterschool programs and two school 


districts in one California community. Outcomes for participants are compared to those for non-participants. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. The study 


was conducted from 2006-09. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome z i ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index 8 8 Grades 5-7 1,05 -0.07 (0.06) 
California Physical : : 
‘ u Tier Ill Tier Ill Grades 5-7 1,05 0.23 (0.06)* 
Fitness Test 
Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 0.08 


* Statistically significant at p<.05 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: London, R. A., & Gurantz, O. (2013). Afterschool program participation, youth physical fitness, and overweight. American Journal of 
Preventive Medicine, 44(s3), S200-S207. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Girls on the Move 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Girls on the Move program, offered in Kentwood and Lan- Location 


sing, Michigan in 2014-15, aimed to increase math achievement LOCALE REGION 
and executive functioning in low-income girls in grades 5-8 

through moderate-to-vigorous physical activity. Programming 

focused on providing enjoyable daily physical activities with a 

choice of active games, dance/aerobics, walking, or sports. The i | 


program was offered for 17 weeks, three days per week. 


Urban Midwest 

=I od ot = OM BAYA) 0-3-0 Wc] Wed ais 

Participants 

CLASS GRADE TARGET STUDENTS 
Positive Effect 

Grades 5-8 Low-Income Girls 
The study found a mix of improved outcomes and null or 
negative outcomes for 
Mathematics Achievement SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Mixed Effects 
The study found a mix of improved outcomes and null or 
negative outcomes for 

Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


9 = 
ment, Other Achievement, Promotion & Graduation, Reading/ELA 81% Low-Income Not Reported 
Achievement, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Sports SETTING: School site 
BEST PRACTICES USED: Active learning DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 17 weeks, 3 days/week 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | GIRLS ON THE MOVE 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of Girls on the Move (GOTM) for girls in grades 5-8 in Kentwood and Lansing, 
Michigan. Schools were randomized to receive GOTM or to serve as a control. Outcomes for participants are compared to 

those for non-participants. The comparison students experienced “business as usual,” which could have included other school- 
and community-sponsored afterschool activities. The study was conducted in 2014-15. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome . : a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index > ) Grades 5-8 150 -0.02 (0.16) 
Maximal oxygen 
wd 0 0 Grades 5-8 150 -0.17 (0.16) 
consumption 
Moderate-to-vigorous A 
; a Tier | Tier Ill Grades 5-8 150 -0.33 (0.16)* 
physical activity 
Percent body fat > ) Grades 5-8 150 0.18 (0.16) 
Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 0.08 


) Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


* Statistically significant at p<.05 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome E : wae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Woodcock-Johnson Ill - . ' 
, Tier | Tier Ill Grades 5-8 150 0.34 (0.16)* 
applied problems 
Woodcock-Johnson IIl - . . 
Tier | Tier Ill Grades 5-8 150 1.20 (0.18)* 
math fluency 
Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 0.77 


* Statistically significant at p<.05 


CITATION: da Cruz, K. (2017). Effects of a randomized trial afterschool physical activity club on the math achievement and executive functioning of 
girls (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 10252421) 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Student-Centered Physical Activity Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The student-centered physical activity program, offered in one Location 
rural South Carolina school, aimed to increase the amount LOCALE 
of physical activity experienced by low-income underserved 
students ages 10-12, as well as their motivation for such 
activity. The program allowed participants a choice of physical 
activities and included motivational and behavioral skills train- 
ing to increase their physical activity outside of the program. 
The program also offered homework help. The program was Reval 
offered for four weeks, three days per week, two hours each 

day. The implementation year is not reported. 


Participants 


CLASS GRADE 


= od od Od AYA | SWC Grade 6 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 
Promotion & Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


SOCIO-ECONOMIC STATUS 


83% Low-Income 


REGION 


Southeast 


TARGET STUDENTS 


Low-Income 


RACE OR ETHNICITY 


83% Black 


Program Content Logistics 


MAIN COMPONENTS: Sports, positive identity/self- SETTING: School site 
concept activities, homework help 


BEST PRACTICES USED: Sequenced, active learning 


DELIVERY FORMAT: Group format 


STAFFING: Program staff, specialized staffing 


DURATION: 4 weeks, 3 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Provider training 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | STUDENT-CENTERED PHYSICAL ACTIVITY PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool physical activity program for sixth grade students from a rural South 
Carolina middle school who volunteered to attend the program. Outcomes for participants are compared to outcomes for sixth grade 
students from a nearby middle school. The comparison students experienced “business as usual,” which could have included other 
school- and community-sponsored afterschool activities. The implementation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome A i oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Moderate-to-vigorous : A 
£ Tier Ill Tier Ill Grade 6 48 1.15 (0.32) * 


physical activity 
Average effect size for Physical Activity/Health outcomes at Tiers I-lIll 


* Statistically significant at p<.05 


CITATION: Wilson, D. K., Evans, A. E., Williams, J., Mixon, G., Sirard, J. R., & Pate, R. (2005). A preliminary test of a student-centered intervention on 
increasing physical activity in underserved adolescents. Annals of Behavioral Medicine, 30, 119-124. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Afterschool Exercise Program 


ea COXCH NP) 109-4 ed STUDY LOCATION AND PARTICIPANTS 


The afterschool exercise program, offered in the Augusta, Location 
Georgia area from 2003-05, engaged healthy but overweight LOCALE REGION 
children in aerobic exercise for 20-40 minutes per day for 

about 15 weeks. Exercise activities, selected based on ability to 

elicit a heart rate of more than 150 beats per minute, included 

running games, jump rope, soccer, and basketball. The Hy | 

program was staffed by researchers who were studying the 


program and was overseen by a physical educator. 


Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 
Healthy but 
EFFECTIVENESS AT A GLANCE Ages 7-11 Overweight Salas 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
PEMEVEMENS Mathematics PenieveMment ouet Achievement, Not Reported 53% Black 
Promotion & Graduation, Reading/ELA Achievement, 

Attendance & Enrollment, School Engagement, Science 47% White 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Sports SETTING: Community site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 14 to 15 weeks 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | AFTERSCHOOL EXERCISE PROGRAM 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of an aerobic exercise program for students ages 7-11 in the Augusta, Georgia 
area. Outcomes for participants are compared to those who did not participate in the program. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study was conducted 
from 2003-05. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome R ; on, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
; Cannot 
Body mass index - z 
ly de ) ) Ages 7-11 100 determing 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill Sanne 


determine 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Davis, C. L., Tkacz, J., Gregoski, M., Boyle, C. A., & Lovrekovic, G. (2006). Aerobic exercise and snoring in overweight children: A randomized 
controlled trial. Obesity, 14(11), 1985-1991. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Scouting Nutrition & Activity Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Scouting Nutrition & Activity Program, offered in seven Location 
Girl Scout troops in the Midwest in 2007-08, aimed to increase LOCALE REGION 
physical activity, healthy eating, and obesity-prevention 
behaviors for fourth and fifth grade girls. The program took 
place in regular Girl Scout troop meetings and included edu- 
cation sessions on healthy eating habits based on a Girl Scout 
curriculum and physical activities. Girls were given badge 
assignments to complete at home with parent assistance. The Le aoe es iiadwest 
program was offered for six months, every week or every other 
week, for 1-2 hours per session. 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 4-5 Girl Scouts 
No Effect SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


j ‘ : ; 28% Low-Income 79% White 
Promotion & Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Sports SETTING: Community site 
BEST PRACTICES USED: Family engagement DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 6 months, weekly or bi-monthly 1-2 hour sessions 


IMPLEMENTATION SUPPORT: Provider training, 
ongoing support 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | SCOUTING NUTRITION & ACTIVITY PROGRAM 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial involved seven Girl Scout troops, three randomly assigned to experience the program and four 
assigned to the comparison group. Outcomes for participants are compared to those who did not participate in the program. The 
comparison students experienced “business as usual,” which could have included other school- and community-sponsored afterschool 
activities. The study was conducted in 2007-08. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ae, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index > ) Grades 4-5 72 0.00 (0.24) 
GE ti f 
heey eee 0 0 Grades 4-5 72 -0.33 (0.24) 
beverages 
Fe alae ) ) Grades 4-5 72 -0.06 (0.24) 
and vegetables 
Eati hil hi 
Sel ia alle 0 0 Grades 4-5 72 -0.42 (.0.24) 
television 
Moderate-to-vi 
on eee ) ) Grades 4-5 72 0.39 (0.24) 


physical activity 


Average effect size for Physical Activity/Health outcomes at Tiers I-III | 000 | 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Rosenkranz, R. R., Behrens, T. K., & Dzewaltowski, D. A. (2010). A group-randomized controlled trial for health promotion in Girl Scouts: 
Healthier troops in a SNAP (Scouting Nutrition & Activity Program). BMC Public Health, 10(1), 81-93. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


GEMS 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The GEMS program, offered to youth in Oakland and Palo Location 
Alto, California, aimed to reduce television viewing and LOCALE REGION 
prevent weight gain among overweight Black girls ages 8-10. 

Participants had a Body Mass Index 2 50th percentile and at 

least one parent/guardian who was overweight. The program 

consisted of daily one-hour dance classes with the aim of Hy | 

encouraging physical activity and fostering weight reduction. 


The intervention also included homework time and discussion 


Urban West 

about the meaning of dance in African American culture. The 
program was offered for 12 weeks, five days per week, 2.5 
hours each day. The implementation year is not reported. 

Participants 

CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Ages 8-10 Black Girls 
No Effect SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
EIMEVEMEDS, Mathematics Pehievement ouet Achievement, 72% with household 100% Black 
Promotion & Graduation, Reading/ELA Achievement, 


Attendance & Enrollment, School Engagement, Science income <$40,000 
Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: Dance SETTING: Community site 

BEST PRACTICES USED: Positive adult-student DELIVERY FORMAT: Group format 


lationships, famil tt t fe I 
relationships, family engagement, respect for student cultures STAFFING: Volunteers 


DURATION: 12 weeks, 5 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: Staff-student cultural similarity 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | GEMS 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of the GEMS program for Black girls with a BMI > 50th percentile and at least 
one parent/guardian who was overweight. Students in the comparison group participated in a health education program focused on 
promoting healthful diet and activity patterns. The study was conducted with youth from Oakland and Palo Alto, California. The imple- 
mentation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ey, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index 0) ) Ages 8-10 60 0.06 (0.26) 
Caloric intake > ) Ages 8-10 60 -0.14 (0.26) 
Moderate-to-vigorous 
a , 0 Ages 8-10 59 0.12 (0.26) 
physical activity 
Percent of caloric 
rene 6 , Ages 8-10 60 0.05 (0.26) 
Waist circumference > ) Ages 8-10 60 0.04 (0.26) 


Average effect size for Physical Activity/Health outcomes at Tiers I-III | 003 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Robinson, T. N., Killen, J. D., Kraemer, H. C., Wilson, D. M., Matheson, D. M., Haskell, W.L., .. ., Varady, A. (2003). Dance and reducing 
television viewing to prevent weight gain in African-American girls: The Stanford GEMS pilot study. Ethnicity and Disease, 13, S1-65-S1-77. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Girlfriends for KEEPS 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Girlfriends for KEEPS (Keys to Eating, Exercising, Playing Location 
and Sharing) program, offered in three elementary schools LOCALE REGION 
in Minnesota, aimed to reduce obesity in Black girls ages 
8-10 with a BMI of 25 or above. The program sought to 
increase the girls’ participation in moderate-to-vigorous 
physical activity and healthy eating. The program also included 
a family education component that reinforced healthy eating 
and activity messages. The program was offered for 12 weeks, Not Reported Iiiadwest 
two days per week, one hour each day. The implementation 
year is not reported. 
Participants 
CLASS GRADE TARGET STUDENTS 
Black Girls with 
EFFECTIVENESS AT A GLANCE Ages 8-10 BMI>24 
No Effect SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
EIMEVEMEDS, Mathematics Pehievement ouet Achievement, 71% with household 87% Black 
Promotion & Graduation, Reading/ELA Achievement, 


Attendance & Enrollment, School Engagement, Science income <$40,000 13% Other 
Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENTS: Sports SETTING: School site 

BEST PRACTICES USED: Family engagement DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 12 weeks, 5 days/week, 2.5 hours/day 


IMPLEMENTATION SUPPORT: Provider training, ongoing 
support, staff-student cultural similarity 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | GIRLFRIENDS FOR KEEPS 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact on physical activity and nutrition of Girlfriends for KEEPS for girls ages 8-10 with 
a BMI of 25 or above. The comparison group was assigned to participate in three Saturday morning enrichment sessions that did not 
include physical activity or nutrition programming. The implementation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome i : ae. Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index > ) Ages 8-10 53 -0.19 (0.28) 
Caloric intake > ) Ages 8-10 53 0.40 (0.28) 
Frui bl 
baie here cies ) ) Ages 8-10 53 -0.29 (0.28) 
servings/day 
Mod -to-vi 
fav seat ales 0 0 Ages 8-10 53 0.06 (0.27) 
physical activity 
Percent of caloric 
Teese ) ) Ages 8-10 53 0.18 (0.28) 
t b 
sabrina taal ” 0 Ages 8-10 53 -0.24 (0.28) 
servings/day 
Waist 
ae 0 0 Ages 8-10 53 -0.50 (0.28) 


circumference (cm) 


Average effect size for Physical Activity/Health outcomes at Tiers I-III 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Story, M., Sherwood, N. E., Himes, J. H., Davis, M., Jacobs, D. R., Jr, Cartwright, Y., Smyth, M., & Rochon, J. (2003). An afterschool obesity 
prevention program for African-American girls: The Minnesota GEMS pilot study. Ethnicity and Disease, 13, S1-54-S1-64. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Physical Activity Program For Boys 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This physical activity program for boys, offered at five Location 
elementary schools in Georgia, aimed to prevent excessive LOCALE REGION 
weight gain and increase the cardiovascular fitness of Black 
boys in grades 3-5. The program focused on engaging youth 
in vigorous physical activity, physical skills development, and 
toning and stretching. The program was offered for the full 


school year, five days per week, two hours each day. The 


implementation year is not reported. 


Not Reported Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


‘ : ‘ ; Not Reported 100% Black 
Promotion & Graduation, Reading/ELA Achievement, 
Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENTS: Sports, homework help SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers, program staff 
DURATION: Full school year, 5 days/week, 2 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | PHYSICAL ACTIVITY PROGRAM FOR BOYS 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial involved students from five elementary schools in Georgia. Outcomes for those randomized to 
the program are compared to those who were randomized to the comparison group. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The implementation 
year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index > ) Grades 3-5 106 0.01 (0.20) 
Bone mineral density > ) Grades 3-5 106 0.00 (0.20) 
Maximal 
ne 0 0 Grades 3-5 106 -0.03 (0.20) 
consumption 
Percent body fat 0) ) Grades 3-5 106 0.02 (0.20) 
sels 0 0 Grades 3-5 106 0.01 (0.20) 


circumference (cm) 


Average effect size for Physical Activity/Health outcomes at Tiers I-Ill | 000 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Howe, C. A., Harris, R. A., & Gutin, B. (2011). A 10-month physical activity intervention improves body composition in young black boys. 
Journal of Obesity, 2001, 1-8. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 332 


PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


SPARK Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Sports, Play and Active Recreation for Kids (SPARK) Location 
program, offered in a suburban New England school in 2014- LOCALE REGION 
15, aimed to develop the cardiovascular fitness and motor skills 
of fifth grade students. The program was guided by the SPARK 
curriculum, which offered aerobic endurance and pacing 


activities as well as physically active, cooperative games. The 


program was offered for 10 weeks, two days per week, 1.5 


hours each day. 


Suburban Northeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement, 


Promotion & Graduation, Reading/ELA Achievement, Not Reported Not Reported 
Attendance & Enrollment, School Engagement, Science 

Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: Sports SETTING: School site 

BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: 10 weeks, 2 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | SPARK PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates outcomes for three groups of fifth grade students: (1) students who participated in a SPARK 

afterschool program, compared to those who did not participate in SPARK,(2) students who participated in a SPARK afterschool program 
and an in-school SPARK program; and (3) students who participated in the afterschool SPARK program only. The 

study was conducted in 2014-15. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; a Group 
Without site and With site and students (Standard error) 


sample criteria sample criteria 


Afterschool 


PARK + distri 
Mile run time § § is eg 40 0.07 (0.35) 


district phys ed 


Afterschool 
SPARK + in-school 
SPARK vs. 
in-school SPARK 


Mile run time 8 8 55 -0.15 (0.27) 


Afterschool 
SPARK + in-school 
SPARK vs. 
in-school SPARK 


Number of curl-ups 8 8 55 0.31 (0.27) 


Afterschool 
SPARK + district 
Number of curl-ups 8 8 phys ed 40 0.10 (0.35) 
vs. district phys 
ed 


Afterschool 
SPARK + district 
phys ed vs. 


district phys ed 


Number of push-ups 8 8 40 -0.31 (0.35) 


Afterschool 


SPARK + in-school 
Number of push-ups oo oo SPARK vs. 55: -0.20 (0.27) 


in-school SPARK 
Average effect size for Physical Activity/Health outcomes at Tiers I-III 0.08 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Gross, C. (2016). The effect of an evidence based physical education curriculum and/or an after school activity program on cardiovascular 
endurance and fundamental motor skill proficiency in grade 5 (Doctoral dissertation, Plymouth State University). Retrieved from http://digitalcommons. 
plymouth.edu/etd/104 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


Afterschool Physical Activity Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This program, offered at two urban middle schools in the Location 
Mountain West region, aimed to increase students’ daily levels LOCALE REGION 
of physical activity through afterschool football, basketball, 

soccer, and volleyball. Each 50-minute session included a 

warm-up, technique and strategy exercises, and a competition. 

The program was offered for two months, five days per week, Hy | 


50 minutes each day. The implementation year is not reported. 


Urban West 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 7-8 General Population 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 
The study found neither improved nor negative outcomes for 
Physical Activity/Health 
NOT MEASURED FOR THIS INTERVENTION: General 
PEMEVEMENS Mathematics PenieveMment ouet Achievement, Not Reported 82% White 2% Black 
Promotion & Graduation, Reading/ELA Achievement, . 
Attendance & Enrollment, School Engagement, Science 10% Latino 6% Other 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Sports SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Specialized staffing, volunteers 
DURATION: 2 months, 5 days/week, 50 minutes/day 


IMPLEMENTATION SUPPORT: None reported 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | AFTERSCHOOL PHYSICAL ACTIVITY PROGRAM 


Study Details 


STUDY DESCRIPTION 

In this randomized controlled trial, seventh and eighth grade students in two urban public schools who volunteered to participate in 

the study were randomlly selected to participate in the program or to the comparison group. The comparison students experienced 
“business as usual,” which could have included other school- and community-sponsored afterschool activities. The study authors do not 
report the implementation year. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome . ; es, Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Physical Activity 
Self-Efficacy Scale Cannot 
) ) Grades 7-8 98 ; 
(Bartholomew, Loukas, determine 


Jowers, & Allus, 2006) 


Physical Activities Cc t 

. ‘ anno 
Questionnaire for ) 0 Grades 7-8 98 determine 
Children (PAQ-C) 


Average effect size for Physical Activity/Health outcomes at Tiers I-III earner 


determine 


. Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Huang, C., Gao, Z., Hannon, J. C., Schultz, B., Newton, M., & Jenson, W. (2012). Impact of an afterschool physical activity program on youth’s 
physical activity correlates and behavior. [CHPER-SD Journal of Research, 7(2), 18-23. 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS 


COPE TEEN 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


COPE TEEN, offered in two suburban high schools in New York Location 
state, aimed to help overweight teens lose weight and develop LOCALE REGION 
healthy behaviors and lifestyles. The intervention consisted of 
cognitive behavioral skill-building as well as physical activity. 
The curriculum that focused on topics including nutrition 
education, emotional and behavioral regulation, self-esteem, 
coping with stress, goal setting and problem solving. The 
program also educated parents on ways to support their Le aoe es Nordieact 
children’s weight loss and healthy behavior. The program was 
offered over nine weeks, 1-2 days per week, for up to 1.5 hours 
each day. The implementation year is not reported. 

Participants 


CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Ages 15-18 Overweight Teenagers 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Physical Activity/Health 


NOT MEASURED FOR THIS INTERVENTION: General 
EIMEVEMEDS, Mathematics Pehicvement ouet Achievement, 70% with household 92% White 
Promotion & Graduation, Reading/ELA Achievement, 


Attendance & Enrollment, School Engagement, Science incomes <$40,000 8% Black 
Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: None reported SETTING: School site 

BEST PRACTICES USED: Active learning, DELIVERY FORMAT: Group format 


famil 
amily engagement STAFFING: Not reported 


DURATION: 9 weeks: 2 days/week for 6 weeks, 
1 day/week for 3 weeks, 1-1.5 hours/day 


IMPLEMENTATION SUPPORT: Provider training 
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PHYSICAL ACTIVITY AND HEALTH PROGRAMS | COPE TEEN 


Study Details 


STUDY DESCRIPTION 

This study estimates the impact of COPE TEEN for students ages 15-18 with BMI of at least 25. Students were randomly assigned to 
participate in the program or to the comparison group. The comparison students experienced “business as usual,” which could have 
included other school- and community-sponsored afterschool activities. The study was conducted in two high schools in upstate New 
York. The implementation year is not reported. 


PHYSICAL ACTIVITY/HEALTH 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ey Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Body mass index 8 8 Ages 15-18 11 0.79 (0.63) 
Weight § § Ages 15-18 "1 0.55 (0.62) 
Average effect size for Physical Activity/Health outcomes at Tiers I-Ill 0.67 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Melnyk, B. M., Small, L., Morrison-Beedy, D., Strasser, A., Spath, L., Kreipe, R., . .. O’Haver, J. (2007). The COPE healthy lifestyles teen 
program: Feasibility, preliminary efficacy, and lessons learned from an after school group intervention with overweight adolescents. Journal of Pediatric 
Health Care, 21(5), 315-322. 
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STEM PROGRAMS 


4-H Robotics Program 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The 4-H Robotics Program, offered in a small rural elementary Location 
school in Nebraska in 2005, aimed to increase the science, LOCALE REGION 
engineering, and technology achievement of rural youth ages 
9-11. The program was staffed by certified teachers and used 
the National 4-H robotics curriculum, which was based on the 
LEGO Mindstorms curriculum. Using experiential learning, the 
program began with basic programming and building tasks and 
advanced to more complex projects. The program was offered Reval Midwest 
for six weeks, two days per week, one hour each day. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Ages 9-11 Low-Income 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Science Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical 


Activity/Health, Promotion & Graduation, Reading/ELA Achievement, Not Reported Not Reported 
Attendance & Enrollment, School Engagement, and Social & Emotional 

Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: STEM activities SETTING: School site 

BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: 6 weeks, 2 days/week, 1 hour/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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STEM PROGRAMS | 4-H ROBOTICS PROGRAM 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool robotics program in a small rural elementary school in Nebraska during 
2005. The comparison group is comprised of students from the school who did not participate in the program. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome i y ret Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Evaluator-created 
assessment of 
science, engineering, . : 
g g Tier Il Tier Il Ages 9-11 32 0.98 (0.38)* 
and technology 
knowledge (general 
and domain-specific) 
Average effect size for Science Achievement outcomes at Tiers I-Ill 0.98 


* Statistically significant at p<.05 


CITATION: Barker, B. S., & Ansorge, J. (2007). Robotics as means to increase achievement scores in an informal learning environment. Journal of 
Research on Technology in Education, 39(3), 229-243. 
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STEM PROGRAMS 


Bringing Up Girls in Science (BUGS) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Bringing Up Girls in Science (BUGS), offered in a Texas Location 
community in 2001-02, aimed to increase the science LOCALE REGION 
achievement of fourth and fifth grade girls through 
environmental science experiences and female mentoring. 
The program used the Science, Technology, and Children 
Curriculum, which provides materials and resources for 


hands-on science experiments. Students were also paired 


with a high school mentor with whom they communicated 


Southwest 
regularly. The program was led by university faculty and staff 
and involved regular campus visits. The program was offered for 
the full school year, one day per week, two hours each day. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 4-5 Girls 


SOCIO-ECONOMIC STATUS’ RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 

Science Achievement 
NOT MEASURED FOR THIS INTERVENTION: General 


Achievement, Mathematics Achievement, Other Achievement, 


Physical Activity/Health, Promotion and Graduation, Reading/ELA Noe Peported Nae Repose 
Achievement, Attendance & Enrollment, School Engagement, and 

Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: STEM activities SETTING: School site 

BEST PRACTICES USED: Sequenced, active learning, DELIVERY FORMAT: Group and one-to-one formats 


ositive adult-student relationships 
peat p . ip STAFFING: Program staff, specialized staffing 


DURATION: Full school year, 1 day/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Program manual 
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STEM PROGRAMS | BRINGING UP GIRLS IN SCIENCE (BUGS) 


Study Details 


STUDY DESCRIPTION 

The study uses a quasi-experimental design to estimate the impact of this afterschool environmental science program. Outcomes 

for participating fourth and fifth grade girls from a mid-sized urban community in northern Texas are compared with outcomes from a 
matched group of girls from another large school district in northern Texas. The comparison students experienced “business as usual,” 
which could have included other school- and community-sponsored afterschool activities. The study was conducted in 2001-02. 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome r : ae: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
lowa Test of Basic Skills : . 
Tier Il Tier Ill Grade 4-5 66 0.65 (0.25)* 


in Science (ITBS-S) 
Average effect size for Science Achievement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


CITATION: Tyler-Wood, T., Ellison, A., Lim, O., & Periathiruvadi, S. (2012). Bringing Up Girls in Science (BUGS): The effectiveness of an afterschool 
environmental science program for increasing female students’ interest in science careers. Journal of Science Education and Technology, 21(1), 46-55. 
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STEM PROGRAMS 


The Investigators Club 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The Investigators Club, offered at one middle school in the Location 
Northeast, aimed to increase the academic motivation of LOCALE REGION 
low-income urban seventh graders through scientific explo- 
ration. The program, which was staffed by certified teachers 
and used The Investigators Club curriculum, engaged 
participants in solving scientific problems presented by the 


program leader. Students experienced hands-on activities, 


experimentation, small-group discussion, and community- 


Northeast 
building activities. The program was offered for 15 weeks, 
three days per week, 1.5 hours each day. The implementation 
year is not reported. 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grade 7 Low-Income 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


School Engagement 


Science Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


: i : é 62% Low-Income 34% White 11% Asian 
ment, Mathematics Achievement, Other Achievement, Physical . 
Activity/Health, Promotion & Graduation, Reading/ELA Achievement, 31% Latino 2% Other 
Attendance & Enrollment, and Social & Emotional Competencies 21% Black 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: STEM activities SETTING: Not reported 
BEST PRACTICES USED: Sequenced, active learning DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: 15 weeks, 3 days/week, 1.5 hours/day 


IMPLEMENTATION SUPPORT: Program manual 
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STEM PROGRAMS | THE INVESTIGATORS CLUB 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of The Investigators Club for seventh grade students in one middle school. Pairs 
of interested students were matched on demographics and science grades, and one student within each pair was randomly selected to 
participate in the program. Outcomes for students who were randomized to participate in the program were compared to outcomes for 
students who were randomized to the comparison group. The comparison students experienced “business as usual,” which could have 
included other school- and community-sponsored afterschool activities. The implementation year is not reported. 


SCHOOL ENGAGEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome z 5 ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Teacher report of 
student engagement : 4 
oe Tier | Tier III Grade 7 54 0.58 (0.28)* 


in science (Connell & 
Wellborn, 1991) 


Average effect size for School Engagement outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Science course grade Tier | Tier Ill Grade 7 54 0.97 (0.29)* 
Average effect size for Science Achievement outcomes at Tiers I-Ill 0.97 


* Statistically significant at p<.05 


CITATION: Grolnick, W. S., Farkas, M. S., Sohmer, R., Michaels, S., & Valsiner, J. (2007). Facilitating motivation in young adolescents: Effects of an 
afterschool program. Journal of Applied Developmental Psychology, 28, 332-344. 
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MULTI-STUDY PROGRAM SUMMARY 


FIRST Robotics 


PROGRAM DESCRIPTION EFFECTIVENESS AT A GLANCE 


FIRST Robotics was a robotics competition for 
high school students designed to develop student 
interest in science, technology, engineering, and 
mathematics. The competition gave high school 


Positive Effect 


Overall, the studies found at least one improved 
outcome with no overriding contrary evidence for 


Attendance & Enrollment 


robotics teams a six-week period to design and 
program a robot to complete a specific task. No Effect 


Following the six-week period, FIRST Robotics Overall the studies foundinewher 


hosted a regional event. : : 
improved nor negative outcomes for 


General Achievement 
Mathematics Achievement 
Reading/ELA Achievement 


School Engagement 


This review of FIRST Robotics identified two studies of this program. The findings from each of these 
studies are presented individually on pages 347 — 351 of this document. 


« Koumoullos, M. (2013). The academic differences between students involved in school-based robotics programs 
and students not involved in school-based robotics programs (Doctoral dissertation). Retrieved from ProQuest 
Dissertations and Theses Database (UMI No. 3573701) 


« Welch, A., & Huffman, D. (2011). The effect of robotics competitions on high school students’ attitudes toward 


science. School Science and Mathematics, 111(8), 416-424. 


MULTI-STUDY PROGRAM SUMMARY FIRST ROBOTICS | SUMMARY #1 


FIRST Robotics (Career & technical school) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This FIRST Robotics program, offered at a career and technical Location 


school in the Northeast in 2011-12, was a robotics competition LOCALE REGION 
for high school students designed to develop student interest 
in science, technology, engineering, and mathematics. The 
competition gave high school robotics teams a six-week period 
to design and program a robot to complete a specific task. 


Following the six-week period, FIRST hosted a regional event. 


Northeast 
EFFECTIVENESS AT A GLANCE 
Participants 
Positive Effect CLASS GRADE TARGET STUDENTS 
The study found at least one improved outcome with no 
overriding contrary evidence for Grades 9-12 General Population 


Attendance & Enrollment 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


No Effect 


The study found neither improved nor negative outcomes for 
General Achievement 


Mathematics Achievement 


Reading/ELA Achievement 


% Low-I % Lati 10% i 
NOT MEASURED FOR THIS INTERVENTION: Other Achievement, EEO Lene ncome So Beane 0% White 
Physical Activity/Health, Promotion & Graduation, School Engagement, 23% Asian 7% Black 
Science Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: STEM activities SETTING: School site 
BEST PRACTICES USED: Active learning DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: 6 weeks 


IMPLEMENTATION SUPPORT: Program manual 
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MULTI-STUDY PROGRAM SUMMARY FIRST ROBOTICS | SUMMARY #1 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of an afterschool robotics program in a large, urban career and technical education 
high school during 2011-12. The comparison group was comprised of students at the school who did not participate in the program. 
The comparison students experienced “business as usual,” which could have included other school- and community-sponsored 
afterschool activities. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome a e a Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Percentage of school 
days attended - January oo co Grades 9-12 69 0.19 (0.25) 
2012 report card 


Percentage of school 
days attended - June Tier Ill Tier Ill Grades 9-12 65 0.62 (0.26) * 
2012 report card 


Average effect size for Attendance & Enrollment outcomes at Tiers I-Ill 


* Statistically significant at p<.05 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


Average effect size for Boce SUISSE ; 
: Number of Effect size 

General Achievement . 5 cow Group 

i Without site and With site and students (Standard error) 
outcomes at Tiers I-Ill avs are 

sample criteria sample criteria 

GPA - January 2012 

is § § Grades 9-12 69 0.17 (0.25) 

report card 
GPA - June 2012 
8 8 Grades 9-12 65 0.00 (0.25) 


report card 


Average effect size for General Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 
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MULTI-STUDY PROGRAM SUMMARY FIRST ROBOTICS | SUMMARY #1 


MATHEMATICS ACHIEVEMENT 


Mathematics course 
grade - June 2012 § 8 Grades 9-12 65 -0.24 (0.25) 
report card 


Mathematics course 
grade - January 2012 8 8 Grades 9-12 69 -0.14 (0.25) 
report card 


State assessment - 


ranthe mates 8 8 Grades 9-12 60 0.12 (0.27) 


Average effect size for Mathematics Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier IIl because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


English course grade - 


June 2012 report card 8 8 Grades 9-12 65 -0.35 (0.26) 
English course 
grade - January 2012 § 8 Grades 9-12 69 -0.17 (0.25) 


report card 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier IIl because there is not a statistically significant improvement in the outcome. 


CITATION: Koumoullos, M. (2013). The academic differences between students involved in school-based robotics programs and students not involved 
in school-based robotics programs (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3573701) 
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MULTI-STUDY PROGRAM SUMMARY FIRST ROBOTICS | SUMMARY #2 


FIRST Robotics (Nine Schools) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This FIRST Robotics program, offered in nine high schools in Location 


a Midwestern metropolitan area, was a robotics competition LOCALE REGION 


for high school students designed to develop student interest 


in science, technology, engineering, and mathematics. The 
competition gave high school robotics teams a six-week period 
to design and program a robot to complete a specific task. 


Following the six-week period, FIRST hosted a regional event. 


The implementation year is not reported. 


Suburban, Urban Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 
EFFECTIVENESS AT A GLANCE Grades 9-12 General Population 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


School Engagement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 
ment, Mathematics Achievement, Other Achievement, Physical 


9 A 9 7 
Activity/Health, Promotion and Graduation, Reading/ELA Achievement, Nor Peported re Waite site coaua 
Attendance & Enrollment, Science Achievement, and Social & 17% Black 2% Other 
Emotional Competencies 3% Latino 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: STEM activities SETTING: School site 
BEST PRACTICES USED: Active learning DELIVERY FORMAT: Group format 


STAFFING: Certified teachers, volunteers 
DURATION: 6 weeks 


IMPLEMENTATION SUPPORT: Program manual 
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MULTI-STUDY PROGRAM SUMMARY FIRST ROBOTICS | SUMMARY #2 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates impact of an afterschool robotics competition for students in nine high schools in a large Midwest- 
ern metropolitan area. Comparison group students, who did not participate in the program, attended the same schools and science 
classes as the students on the robotics team. The comparison students experienced “business as usual,” which could have included 
other school- and community-sponsored afterschool activities. The implementation year is not reported. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome Group 


Without site and With site and students (Standard error) 
sample criteria sample criteria 


Test of Science 
Related Attitudes 
(TOSRA) - Leisure 
interest in science 


8 8 Grades 9-12 99 0.36 (0.21) 


Test of Science 
Related Attitudes 
(TOSRA) - Adoption 
of scientific attitudes 


8 8 Grades 9-12 99 0.34 (0.21) 


Test of Science 
Related Attitudes 
(TOSRA) - Attitude 
to scientific inquiry 


8 8 Grades 9-12 99 0.33 (0.21) 


Test of Science 
Related Attitudes 
(TOSRA) - Career 
interest in science 


8 8 Grades 9-12 99 0.33 (0.21) 


Test of Science 
Related Attitudes 
(TOSRA) - Enjoyment of 
science lessons 


8 8 Grades 9-12 99 0.35 (0.21) 


Average effect size for School Engagement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Welch, A., & Huffman, D. (2011). The effect of robotics competitions on high school students’ attitudes toward science. School Science and 
Mathematics, 111(8), 416-424. 
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STEM PROGRAMS 


FIRST Lego League 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The FIRST Lego League (FLL), offered in one charter school Location 
in 2013, was an afterschool robotics program that aimed LOCALE REGION 
to develop student interest in engineering and math and 

improve math skills. Working in teams, students in grades 4-6 


designed, created, and programmed a robot to solve a set of 


problems using the LEGO Mindstorms Invention set. Teacher 
coaches supported team-building through activities and 


resources provided by FLL. 


Not Reported West 
Participants 
CLASS GRADE TARGET STUDENTS 

EFFECTIVENESS AT A GLANCE Grades 4-5 Gifted/Talented Students 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 

No Effect 

The study found neither improved nor negative outcomes for 

Mathematics Achievement 

NOT MEASURED FOR THIS INTERVENTION: General Achieve- 

ment, Other Achievement, Physical Activity/Health, Promotion & 

Graduation, Reading/ELA Achievement, Attendance & Enrollment, Not Reported Not Reported 

School Engagement, Science Achievement, and Social & Emotional 

Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: STEM activities SETTING: School site 

BEST PRACTICES USED: Active learning DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: Program manual, 
ongoing support 
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STEM PROGRAMS | FIRST LEGO LEAGUE 


Study Details 


STUDY DESCRIPTION 

This study uses a quasi-experimental design to estimate the impacts of an afterschool robotics program on mathematics achievement 
among elementary school students attending a charter school in 2013. The study compares outcomes for fourth and fifth grade students 
who participated in the program to outcomes for a matched group of non-participants. The comparison students experienced “business 
as usual,” which could have included other school- and community-sponsored afterschool activities. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome r : Ree Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Renaissance Star Math oo co Grade 5 46 -0.26 (0.30) 
Renaissance Star Math oo co Grade 4 46 -0.18 (0.30) 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 


co Meets Tier Il standards for research quality. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


CITATION: Laughlin, S. R. (2013). Robotics: Assessing its role in improving mathematics skills for grades 4 to 5 (Doctoral dissertation). Retrieved from 
ProQuest Dissertations and Theses Database (UMI No. 3557417) 
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STEM PROGRAMS 


Leonardo’s Laboratory 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


Leonardo’s Laboratory, offered in one elementary school in Location 
New Jersey during 2006-07, aimed to improve the overall LOCALE REGION 
classroom behavior and performance of urban students in 
grades 3-5 who were gifted in the area of visual spatial skills 
but were diagnosed with a learning disability. The program 
engaged students in a series of eight engineering projects 
based on Leonardo DaVinci’s designs. The projects required 
planning, time management, follow-through, and exploration. Sian Nordieact 
The program was offered for eight weeks, one day per week, 


two hours each day. 


Participants 


CLASS GRADE TARGET STUDENTS 


Gifted, With 
EFFECTIVENESS AT A GLANCE Grades 3-5 emnine Dcabiity 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
No Effect 


The study found neither improved nor negative outcomes for 


Social & Emotional Competencies 


NOT MEASURED FOR THIS INTERVENTION: General 
Achievement, Mathematics Achievement, Other Achievement 7 : 

: d : Not Reported 73% Latino 4% Asian 
Physical Activity/Health, Promotion and Graduation, Reading/ELA P : 
Achievement, Attendance & Enrollment, School Engagement, and 17% White 4% Black 
Science Achievement 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: STEM activities SETTING: School site 
BEST PRACTICES USED: Sequenced, respect for DELIVERY FORMAT: Group format 


It 
STHGe rie culsures STAFFING: Specialized staffing 


DURATION: 8 weeks, 1 day/week, 2 hours/day 


IMPLEMENTATION SUPPORT: Program manual, 
provider training 
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STEM PROGRAMS | LEONARDO’S LABORATORY 


Study Details 


STUDY DESCRIPTION 

This randomized controlled trial estimates the impact of Leonardo’s Laboratory for students in grades 3-5 who had above average 
visual-spatial skills as measured by the Culture Fair Intelligence Test and were identified by their teachers as struggling with the 
traditional curriculum. Outcomes for students who were randomized to the program are compared to outcomes for students randomized 
to the comparison group. The comparison students experienced “business as usual,” which could have included other school- and 
community-sponsored afterschool activities. The study was conducted at one elementary school in Passaic, New Jersey during 2006-07. 


SOCIAL & EMOTIONAL COMPETENCIES 


ESSA evidence tier 
Number of Effect size 
Outcome . ; ma Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Teacher Rating Scale 
of the Behavior 
Assessment System 
u 0 0 Grades 3-5 23 -0.13 (0.42) 


for Children 
(BASC-2) - Attention 
Problems scale 


Teacher Rating Scale 
of the Behavior 
Assessment System ) ) Grades 3-5 23 0.03 (0.42) 
for Children (BASC-2) - 
Leadership Scale 


Teacher Rating Scale 
of the Behavior 
Assessment System > ) Grades 3-5 23 -0.15 (0.42) 
for Children (BASC-2) - 
Social Skills scale 


Average effect size for Social & Emotional Competencies outcomes at Tiers I-lll -0.08 


0 Meets Tier | standards for cause-and-effect. Does not meet Tier | because there is not a statistically significant improvement in the outcome. 


CITATION: Lipton, C. M. (2007). A domain-specific approach to giftedness: An empirical study of the effects of an enrichment program on students’ 
leadership, social skills, and attention (Doctoral dissertation). Retrieved ProQuest Dissertations and Theses Database (UMI No. 3269342) 
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STEM PROGRAMS 


STEM Excellence and Leadership 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The STEM Excellence and Leadership program, offered in 11 Location 

rural districts in lowa, aimed to prepare academically success- LOCALE REGION 

ful, rural, low-income students in grades 6-8 for advanced 

high school coursework in STEM. The program offered 

additional math and science course work and provided 

professional development for teachers in the areas of math, 

science, and gifted education. The program was offered for 

24 weeks, with an average participation of 96 hours. Raval Midwest 
Participants 
CLASS GRADE TARGET STUDENTS 

EFFECTIVENESS AT A GLANCE Grades 6-8 High-Performing 
SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 

No Effect 


The study found neither improved nor negative outcomes for 


Mathematics Achievement 


Science Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


0, a 
ment, Other Achievement, Physical Activity/Health, Promotion and 48% Low-Income Not Reported 
Graduation, Attendance & Enrollment, School Engagement, Science 
Achievement, and Social & Emotional Competencies 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: STEM activities SETTING: School site 
BEST PRACTICES USED: None reported DELIVERY FORMAT: Not reported 


STAFFING: Certified teachers 
DURATION: 24 weeks, average participation=96 hours 


IMPLEMENTATION SUPPORT: Provider training 
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STEM PROGRAMS | STEM EXCELLENCE AND LEADERSHIP 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of the STEM Excellence and Leadership program for high-achieving students in grades 
6-8. Outcomes for program participants are compared to outcomes for students who did not participate. The comparison students 
experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. The study 


was conducted in 11 schools in 11 rural districts in lowa. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; : ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
ACT Explore - math 8 8 Grades 6-8 123 0.28 (0.18) 


Average effect size for Mathematics Achievement outcomes at Tiers I-Ill 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCIENCE ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ‘ ; oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
ACT Explore - science 8 8 Grades 6-8 123 0.00 (0.18) 


Average effect size for Science Achievement outcomes at Tiers I-Ill 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION:Assouline, S. G., Ihrig, L. M., Mahatmya, D. (2017). Closing the excellence gap: Investigation of an expanded talent search model for student 
selection into an extracurricular STEM program in rural middle schools. Gifted Child Quarterly, 61, 250-261. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 357 


Sports Programs 


SPORTS PROGRAMS 


School-Sponsored Sports (Texas) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The program consisted of participation in school-sponsored Location 
sports in three districts in Texas. The implementation year is LOCALE REGION 
not reported. 
Urban Southwest 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grade 8 General Population 
Positive Effect 
The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 


School Engagement 


No Effect 


The study found neither improved nor negative outcomes for 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 66% Low-lacomeé 39% Latino 25% Black 
ment, Mathematics Achievement, Other Achievement, Physical Activity/ 

Health, Promotion & Graduation, Attendance & Enrollment, Science 33% White 3% Other 
Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Sports SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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SPORTS PROGRAMS | SCHOOL-SPONSORED SPORTS (TEXAS) 


Study Details 


STUDY DESCRIPTION 
This quasi-experiment estimates the impact of having participated in school-sponsored sports and performing arts in eighth grade 
on academic achievement and school engagement in ninth grade. The study examines these outcomes for students in three school 


districts in Texas. The implementation year is not reported. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 


Number of Effect size 
Outcome : . Pee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Language arts 
ke § § Grade 8 495 -0.04 (0.09) 


course grade 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


SCHOOL ENGAGEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ’ ; rane Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 

Motivation for 
Education Attainment q ' 
: j Tier Ill Tier Ill Grade 8 495 0.42 (0.09)* 
Questionnaire: Value of 
Education subscale 
Teacher rating of 

2 g 8 Grade 8 495 -0.04 (0.09) 


classroom engagement 


Average effect size for School Engagement outcomes at Tiers I-lll 


* Statistically significant at p<.05 
§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Hughes, J. N., Cao, Q., & Kwok, O. (2016). Indirect effects of extracurricular participation on academic adjustment via perceived friends’ 
prosocial norms. Journal of Youth and Adolescence, 45, 2260-2277. 
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SPORTS PROGRAMS 


High School Interscholastic Sports 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


This quasi-experiment uses data from the National Location 
Educational Longitudinal Study (NELS) to estimate the LOCALE REGION 
impact of participating in high school interscholastic sports 

on student achievement. Outcomes for students who reported 
participating in sports in both tenth and twelfth grades were 


compared to outcomes for students who did not participate 


in sports in either year. 


Rural, Suburban, Urban All Regions 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE 


Positive Effect 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- Average family 75% White 4% Asian 
ment, Other Achievement, Physical Activity/Health, Promotion & income = $42,400 

Graduation, Attendance & Enrollment, School Engagement, Science canes 11% Black 1% Other 
Achievement, and Social & Emotional Competencies 9% Latino 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Sports SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Not reported 


STAFFING: Not reported 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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SPORTS PROGRAMS | HIGH SCHOOL INTERSCHOLASTIC SPORTS 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment uses data from the National Educational Longitudinal Study (NELS) to estimate the impact of participating in high 
school interscholastic sports on student achievement. Outcomes for students who reported participating in sports in both tenth and 
twelfth grades were compared to outcomes for students who did not participate in sports in either year. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Math achievement test esd Rist Cc ; 
4 4 7 7 raqes fe) anno 
-N nalE nal t 
SH a ae a Tier Ill Tier Ill 10-12 Reported determine’ 
Longitudinal Study 
Mathematics . , Grades Not Cannot 
course grade Tent fet 10-12 Reported determine* 


‘ 5 , ‘ Cannot 
Average effect size for Mathematics Achievement outcomes at Tiers I-III 


determine 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome : . ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
: . : Grades Not Cannot 
English course grade : 
9 9 Her on 10-12 Reported determine* 
Reading achievement 
test - National P ‘ Grades Not Cannot 
Educational fen Hen 10-12 Reported determine* 


Longitudinal Study 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-IIl Sonnet 


determine 


* Statistically significant at p<.05 


CITATION: Broh, B. A. (2002). Linking extracurricular programming to academic achievement: Who benefits and why? Sociology of Education, 75, 69-95. 
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SPORTS PROGRAMS 


School-Sponsored Sports (Miami) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


School-sponsored varsity athletics programs served male and Location 
female high school students in Florida in 2002-03. Activities LOCALE REGION 
included any sport recognized by the Florida High School 
Athletics Association. 
Urban Southeast 
Participants 
CLASS GRADE TARGET STUDENTS 


EFFECTIVENESS AT A GLANCE Grades 9-12 General Population 


SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


Positive Effect 


The study found at least one improved outcome with no 
overriding contrary evidence for 


Mathematics Achievement 


Reading/ELA Achievement 


NOT MEASURED FOR THIS INTERVENTION: General Achieve- 


: ; a ‘ Not Reported 60% Latino 
ment, Other Achievement, Physical Activity/Health, Promotion & 
Graduation, Attendance & Enrollment, School Engagement, Science 32% Black 
Achievement, and Social & Emotional Competencies 9% White 
PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 
Program Content Logistics 
MAIN COMPONENT: Sports SETTING: Not reported 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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SPORTS PROGRAMS | SCHOOL-SPONSORED SPORTS (MIAMI!) 


Study Details 


STUDY DESCRIPTION 

For this quasi-experiment, conducted in the Miami-Dade County Public Schools, 10 senior high schools were randomly selected from 
31 comprehensive senior high schools during the 2002-03 school year. From each school, about 100 students participating in school 
sports and 100 non-participants were randomly selected, and their attendance and achievement were compared. Sports participants 
were varsity athletes selected from each school’s Florida High School Athletic Association (FHSAA) athletic eligibility list. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . ‘ er Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Florida Comprehensive 
Assessment Test - Tier Ill Tier Ill Grades 9-12 2,081 0.15 (0.04)* 


mathematics 
Average effect size for Mathematics Achievement outcomes at Tiers I-III pow 


READING/ELA ACHIEVEMENT 


* Statistically significant at p<.05 


ESSA evidence tier 
Number of Effect size 
Outcome . ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Florida Comprehensive 
Assessment Tier Ill Tier Ill Grades 9-12 2,081 0.11 (0.04)* 


Test - reading 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-IIl pon 


* Statistically significant at p<.05 


CITATION: JacAngelo, N. P. (2003). The relation of sports participation to academic performance of high school students (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3126425) 
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SPORTS PROGRAMS 


School-Sponsored Sports (South Texas) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The program consisted of extracurricular sports activities at the Location 


high school level for Latina girls in one urban district in Texas. LOCALE REGION 


The implementation year is not reported. 


EFFECTIVENESS AT A GLANCE 


Positive Effect 


Participants 
CLASS GRADE TARGET STUDENTS 


The study found at least one improved outcome with no 


overriding contrary evidence for 

— Attendan Enrollment 

petendence SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 
General Achievement 

-) Mathematics Achievement 


©) Reading/ELA Achievement 


7 , 
NOT MEASURED FOR THIS INTERVENTION: Other Achievement, Mot Reported han 
Physical Activity/Health, Promotion & Graduation, School Engagement, 

Science Achievement, and Social & Emotional Competencies 

PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 

Program Content Logistics 

MAIN COMPONENT: Sports SETTING: School site 

BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Not reported 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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SPORTS PROGRAMS | SCHOOL-SPONSORED SPORTS (SOUTH TEXAS) 


Study Details 


STUDY DESCRIPTION 
This quasi-experimental study examined the effects of participation in school-sponsored sports for Latina eleventh grade girls in an 
urban district in South Texas. The comparison group was comprised of girls who did not participate in sports. The implementation 


year is not reported. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; re Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Number of school days : : 
¥ Tier Ill Tier Ill Grade 11 788 0.29 (0.08) * 


attended 


Average effect size for Attendance & Enrollment outcomes at Tiers I-III | oz 


* Statistically significant at p<.05 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; oe Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
GPA Tier Ill Tier Ill Grade 11 788 0.19 (0.08) * 


Average effect size for General Achievement outcomes at Tiers I-lll | ow 


* Statistically significant at p<.05 
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SPORTS PROGRAMS | SCHOOL-SPONSORED SPORTS (SOUTH TEXAS) 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Outcome f ; Pape 
Without site and With site and 
sample criteria sample criteria 


Texas Assessment of 
Knowledge and Skills - Tier Ill Tier Ill 
mathematics 


Number of 
Group 
students 
Grade 11 788 


Average effect size for Mathematics Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Outcome : ; ase 
Without site and With site and 
sample criteria sample criteria 
Texas Assessment of 
Knowledge and Skills - Tier Ill Tier Ill 


reading 


Number of 
Group 
students 
Grade 11 788 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


* Statistically significant at p<.05 


Effect size 
(Standard error) 


0.51 (0.08) * 


Effect size 
(Standard error) 


0.38 (0.08) * 


CITATION: Manlove, K. J. (2013). The impact of extracurricular athletic activities on academic achievement, disciplinary referrals, and school attendance 
among Latino female 11th grade students (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3566593) 
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SPORTS PROGRAMS 


SquashSmarts 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


SquashSmarts, offered at two middle schools in Philadelphia, Location 


Pennsylvania in 2008-09, was a sports-based youth LOCALE REGION 
development program that aimed to promote health and 

fitness, academic success, and service by engaging urban 

sixth graders in the game of squash. The program provided 

squash instruction and competitive opportunities coupled with i | 

homework help and individual mentoring. Participants were 


also required to participate in community service activities. The 


Urban Northeast 
program was offered for the full school year, three days per 
week, three hours each day. 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 6-7 General Population 


No Effect 


The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 
School Engagement 
General Achievement 
Mathematics Achievement 


Reading/ELA Achievement 


Not Reported 47% Black 8% Asian 
NOT MEASURED FOR THIS INTERVENTION: Other Achievement, P i 
Physical Activity/Health, Promotion & Graduation, Attendance & Enroll- 35% Latino 10% Other 
ment, Science Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENTS: Sports, homework help SETTING: Community site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Program staff 
DURATION: Full school year, 3 days/week, 3 hours/day 


IMPLEMENTATION SUPPORT: None reported 
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SPORTS PROGRAMS | SQUASHSMARTS 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment compared academic outcomes for students who participated in the SquashSmarts programs to outcomes for a 
matched group of comparison students who did not participate. The comparison students experienced “business as usual,” which could 
have included other school- and community-sponsored afterschool activities. The study was conducted during 2008-09 at two middle 


schools in Philadelphia, Pennsylvania. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; Peet Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
GPA cS} 8 Grades 6-7 61 0.13 (0.26) 


Wechsler Individual 
Achievement Test - oo co Grades 6-7 73 -0.42 (0.24) 
version 2 - total 


Average effect size for General Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome . : noe: Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Wechsler Individual 
Achievement Test - 0 00 Grades 6-7 I3 -0.03 (0.23) 
version 2 - mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-IIl 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 
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SPORTS PROGRAMS | SQUASHSMARTS 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ' ; Ty Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Wechsler Individual 


Achievement Test - co co Grades 6-7 73 -0.15 (0.23) 
version 2 - reading 


Wechsler Individual 
Achievement Test 


; F cd co Grades 6-7 73 -0.38 (0.24) 
- version 2 - written 
language 
Wechsler Individual 
Achi Te 
chievement Test Bs 00 Grades 6-7 73 -0.36 (0.24) 


- version 2 - written 
language 


Average effect size for Reading/ELA Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


SCHOOL ENGAGEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome q fi ee Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
School Success Profile 
- School Engagement 
: (oe) (oe) r -7 7 .04 (0.24 
and Trouble Avoidance Grades 6 S 0.04 (0.24) 
Dimensions - behavioral 
School Success Profile 
- School Engagement 
and Trouble Avoidance 8 8 Grades 6-7 67 0.05 (0.24) 


Dimensions - 
psychological 


Average effect size for School Engagement outcomes at Tiers I-Ill 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier III because there is not a statistically significant improvement in the outcome. 


CITATION: Green, H. K. (2010). The impact of an academic sports-mentoring afterschool program on academic outcomes in at-risk youth (Unpublished 
doctoral dissertation). Philadelphia, PA: Drexel University. 
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SPORTS PROGRAMS 


School-Sponsored Sports (Career & Technical School) 


PROGRAM DESCRIPTION STUDY LOCATION AND PARTICIPANTS 


The program was school-sponsored sports participation in Location 
a large, urban, career and technical education high school LOCALE REGION 
during 2011-12. 
Urban Northeast 
Participants 
EFFECTIVENESS AT A GLANCE CLASS GRADE TARGET STUDENTS 
Grades 9-12 General Population 


No Effect 


The study found at least one improved outcome with no SOCIO-ECONOMIC STATUS RACE OR ETHNICITY 


overriding contrary evidence for 
Attendance & Enrollment 
General Achievement 
Mathematics Achievement 


Reading/ELA Achievement 


% Low-I % Lati 10% Whi 
NOT MEASURED FOR THIS INTERVENTION: Other Achievement, Ee Va neamne ota banne 0% White 
Physical Activity/Health, Promotion & Graduation, School Engagement, 23% Asian 7% Black 
Science Achievement, and Social & Emotional Competencies 


PROGRAM FEATURES AS IMPLEMENTED IN THIS STUDY 


Program Content Logistics 
MAIN COMPONENT: Sports SETTING: School site 
BEST PRACTICES USED: Not reported DELIVERY FORMAT: Group format 


STAFFING: Certified teachers 
DURATION: Not reported 


IMPLEMENTATION SUPPORT: None reported 
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SPORTS PROGRAMS | SCHOOL-SPONSORED SPORTS (CAREER & TECHNICAL SCHOOL) 


Study Details 


STUDY DESCRIPTION 

This quasi-experiment estimates the impact of afterschool sports participation in a large, urban career and technical education high 
school during 2011-12. The comparison group was comprised of students at the school who did not participate in sports. The comparison 
students experienced “business as usual,” which could have included other school- and community-sponsored afterschool activities. 


ATTENDANCE & ENROLLMENT 


ESSA evidence tier 
Number of Effect size 
Outcome g 2 he Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 


Percentage of school 
days attended - January oo co Grades 9-12 74 0.11 (0.23) 


2012 report card 


Percentage of school 
days attended - June 8 8 Grades 9-12 71 0.24 (0.24) 
2012 report card 


Average effect size for Attendance & Enrollment outcomes at Tiers I-III 0.17 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier II because there is not a statistically significant improvement in the outcome. 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


GENERAL ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ’ ras) Group 

Without site and With site and students (Standard error) 

sample criteria sample criteria 

GPA - June 2012 
8 8 Grades 9-12 71 0.06 (0.24) 
report card 
GPA - Janunary 2012 

Q co co Grades 9-12 74 0.04 (0.23) 


report card 


Average effect size for General Achievement outcomes at Tiers I-lll | 00s | 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 
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SPORTS PROGRAMS | SCHOOL-SPONSORED SPORTS (CAREER & TECHNICAL SCHOOL) 


MATHEMATICS ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome 5 ; ea Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
Mathematics course 
grade - January 2012 co co Grades 9-12 74 -0.27 (0.23) 
report card 
Mathematics course 
grade - June 2012 8 8 Grades 9-12 71 -0.26 (0.24) 
report card 
State assessment - 
8 8 Grades 9-12 68 -0.30 (0.24) 


mathematics 


Average effect size for Mathematics Achievement outcomes at Tiers I-III 


co Meets Tier Il standards for cause-and-effect. Does not meet Tier Il because there is not a statistically significant improvement in the outcome. 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


READING/ELA ACHIEVEMENT 


ESSA evidence tier 
Number of Effect size 
Outcome ; ; ae Group 
Without site and With site and students (Standard error) 
sample criteria sample criteria 
English course grade - 
r -12 71 -0.04 (0.24 
June 2012 report card 8 8 eiades'2 Oa (0:24) 
English course grade - 
r -12 74 -0.12 (0.2 
January 2012 report card 8 8 Slades? Cla \t2e) 
Average effect size for Reading/ELA Achievement outcomes at Tiers I-III -0.08 


§ Meets Tier Ill standards for cause-and-effect. Does not meet Tier Ill because there is not a statistically significant improvement in the outcome. 


CITATION: Koumoullos, M. (2013). The academic differences between students involved in school-based robotics programs and students not involved 
in school-based robotics programs (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3573701) 
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APPENDIX EG-1 


STUDIES OF SCHOOL-SPONSORED EXTRACURRICULAR PROGRAMS 


About studies of school-sponsored extracurricular programs 


This appendix describes studies of the effects of participation in school-sponsored extracurricular programs on a range of outcomes. 
Importantly, these studies do not distinguish the effects of participation in different types of activities. Some studies meet Tier Ill, and 
others meet Tier IV. We present studies of extracurricular programs with a paragraph only—rather than a full program summary— 
because though they provide useful information about the effects of a set of afterschool activities, they are not studies of a specific 
program model with defined components. 


STUDIES OF EXTRACURRICULAR ACTIVITIES THAT MEET CAUSE-AND-EFFECT REQUIREMENTS FOR TIER III 


Extracurricular activities (California) 


The quasi-experimental study estimates the effects on high school students’ attitudes toward school and academic aspirations of 
participation in extracurricular activities or other afterschool programs. The study found that students who participated in at least one 
extracurricular activity had higher academic aspirations and more positive attitudes toward school than students who did not participate 
in any extracurricular activities. The study was conducted in six California high schools in 1987-88 and 1989-90. 


Darling, N., Caldwell, L. L., & Smith, R. (2005). Participation in school-based extracurricular activities and adolescent adjustment. Journal 
of Leisure Research, 37(1), 51-76. 


Extracurricular activities (Chicago) 


The quasi-experimental study estimates the effects on behavior, verbal ability, and reading ability of participation in extracurricular 
activities or other afterschool programs, using data at two time points from the Project on Human Development in Chicago 
Neighborhoods survey. At the first time point, when students were 9, 12, or 15 years old, their caregivers were asked whether they 
participated in at least one afterschool activity; students whose caregivers answered affirmatively are characterized as “participants,” 
and others formed the comparison group. At the second time point, three years later, student outcomes were assessed. The study 
found a statistically significant improvement in verbal ability as measured by the Wechsler Intelligence Scale for Children-Revised and 
no effect on behavior or reading ability. The study was conducted in Chicago in 1995 and 1998. 


Isbell, A. M. (2014). Analysis of the relationships between after-school programs/activities and competencies (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3367008) 


Extracurricular activities (Indiana) 


The quasi-experimental study estimates the effects on English language arts and mathematics achievement and school attendance of 
participation in school-based extracurricular activities. The study found statistically significant improvements in English language arts 
and mathematics scores and no differences in attendance. The study was conducted in six rural Indiana high schools in 2010-11. 


Wilcox, M. L. (2012). The impact of extracurricular activities on academic performance for rural secondary students in Indiana (Doctoral 
dissertation). Retrieved from http://scholars.indstate.edu/handle/10484/4006 


Extracurricular activities (Texas) 


This quasi-experimental study estimates the effects on attitudes toward education, ninth grade English language arts (ELA) course 
grade, and classroom engagement (as rated by teachers) of eighth graders’ participation in extracurricular activities other than 
school-sponsored sports. Outcomes for students who participated are compared to outcomes for students who did not participate in 
these activities. The study found a statistically significant improvement in classroom engagement but no differences for attitudes toward 
education or ninth grade ELA course grade. The study was conducted in three Texas school districts in 2008. 


Hughes, J. N., Cao, Q., & Kwok, O. M. (2016). Indirect effects of extracurricular participation on academic adjustment via perceived 
friends’ prosocial norms. Journal of Youth and Adolescence, 45(11), 2260-2277. 
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STUDIES OF EXTRACURRICULAR ACTIVITIES THAT MEET CAUSE-AND-EFFECT REQUIREMENTS FOR ESSA TIER IV 


Extracurricular activities (Missouri) 


This study uses a quasi-experimental design to estimate the effects of school-based extracurricular activities on achievement and 
attendance among elementary school students. Using a sample of students in grades 3-5 in one school, the study compares outcomes 
for students who participated in school-based extracurriculars to outcomes for those who did not participate. This school offered the 
following extracurricular activities: drama club, choir, an ecology/environment activity, science club, chess club, “Girls on the Run,” a 
blended running/physical activity and English/language arts activity, and a math competition group. Each of the activities met once per 
week for 60 minutes. The study was conducted in one Missouri school in 2011-12. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Israel, J. M. (2013). Student extracurricular participation, student achievement, and school perception: An elementary school perspective 
(Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3561144) 


Extracurricular activities and sports (New Jersey) 


This quasi-experiment estimates the effects on GPA and SAT scores of participating in school sports and/or extracurricular activities. 
Outcomes for students who participated in at least one sport or activity were compared to outcomes for students who did not 
participate in any sports or activities. The study was conducted in one New Jersey high school in 2012-13. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Scarpello, J. P. (2014). Examination of the relationship between academic and demographic variables at Hillsborough High School, NJ 
(Doctoral dissertation). Retrieved from http://delaware.contentdm.oclc.org/cdm/compoundobject/collection/p15323coll5/id/26596/rec/1 


Extracurricular activities (Virginia) 


This quasi-experimental study assesses effects on reading and math achievement of participation in school-based extracurricular 
activities. Outcomes for students who participated in at least one activity were compared to outcomes for students who did not 
participate in any activities. The study was conducted in two Virginia middle schools in 2013-14. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Callender, S. L. (2015). Student participation in extracurricular activities and the impact on academic achievement, self-concept, and 
academic self-concept during the middle school years (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses 
Database (UMI No. 3663019) 


Extracurricular activities and sports (Georgia) 


This quasi-experiment estimates the effects on scores on the Georgia High School Graduation Test of participation in school sports, 
band, chorus, or drama for three cohorts of eleventh graders. Outcomes for students who participated in at least one activity were 
compared to outcomes for students who did not participate in any activities. The study was conducted in one Georgia high school in 
2002-04. The study reports that there were no statistically significant effects of the program. The study does not meet requirements for 
a higher evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Moran, R. A. (2009). Extracurricular activities: Investigating the effects of participation-nonparticipation on the Georgia high school 
graduation test (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3350419) 
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Extracurricular activities and sports (Idaho) 


This quasi-experiment estimates the impact of participation in school sports and/or extracurricular activities on the academic 
achievement of twelfth graders. The comparison group did not participate in any sports or activities. The study was conducted 
in two rural schools in Idaho in 2001-02. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Miller, J. S. (2004). An examination of academic achievement and the development of personal attributes of students participating 
in extracurricular activities in two rural Idaho high schools (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses 
Database (UMI No. 3123843) 


Extracurricular activities and sports (NELS study) 


This quasi-experiment uses data from the National Educational Longitudinal Study (NELS) to estimate the impact on mathematics 
achievement of participation in school sports or extracurricular clubs for high school students. Outcomes for students who participated 
in these activities in twelfth grade are compared to outcomes for students who did not participate in any activities. The data are from 
1988-92. 


The study reports improved outcomes, but the eligible outcomes do not meet review requirements. 
Lipscomb, S. (2007). Secondary school extracurricular involvement and academic achievement: A fixed effects approach. Economics 


of Education Review, 26(4), 463-472. 


Extracurricular activities and sports (TIMSS study) 


This quasi-experiment estimates the impact on math and science achievement of fourth graders’ participation in afterschool sports or 
clubs. The data are from Trends in International Mathematics and Science Study (TIMSS) 2011. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Li, Y. (2016). Three essays on economic evaluation of health intervention programs and health policy (Doctoral dissertation). Retrieved 
from https://oaktrust.library.tamu.edu/handle/1969.1/158142 
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APPENDIX EG-2 


STUDIES THAT DO NOT MEET TIER III REQUIREMENTS (TIER IV) 


About studies that do not meet Tier I-III requirements 


This appendix describes studies that compare outcomes for program participants to outcomes for a comparison group of students but 
do not meet all of the Cause-and-Effect requirements for Tiers I-Ill. We refer to these as Tier IV studies. The most typical reason why 
these studies do not meet Tier Ill or above is that they do not include sufficient statistical controls for pre-existing differences between 
program and comparison groups. Studies lacking these statistical controls are not strong tests of program effectiveness, and we have 
low confidence that any observed differences are the result of the program. 


In this appendix, we summarize each Tier IV study so that education decision-makers can have a complete accounting of all publicly 
available comparison group studies of afterschool programs. Evidence from studies with comparison groups is superior to that from 
studies that compare pre- and post-program outcomes for participants only. A Tier IV study with one or more positive effects may 
contribute to a well-specified, research-informed logic model for an afterschool program, as required by ESSA’s Tier IV evidence 
provision. This approach would enable a school or district to implement the afterschool program while examining its impact with a more 
rigorous study that could meet Tier I-Ill requirements. 


STUDIES OF ACADEMIC PROGRAMS 


21st Century Community Learning Centers program for ninth graders 


This quasi-experiment estimates the impact on mathematics and English language arts achievement of an afterschool tutoring program 
for ninth graders who scored below basic on the eighth grade state standardized assessment or were recommended by a teacher as 
being in need of tutoring. Outcomes for students who attended tutoring at least two days per week are compared to outcomes for 
students who did not participate. The study was conducted in two South Carolina high schools during 2013-14. The study reports an 
improved outcome but does not adequately control for pre-existing differences between the program and comparison groups. 


Scott, V. (2014). The impact of an after-school tutorial program on at-risk ninth grade students' academic performance in an Urban 
Midlands School District (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3662524) 


21st Century Community Learning Centers program (Louisville, Mississippi) 

This quasi-experiment estimates the impact on academic achievement of an afterschool program at four schools in the Louisville 
Municipal School District in Mississippi. The program served students in grades 5-8. Programming consisted mainly of academic 
enrichment aimed at increasing vocabulary, reading fluency, comprehension, math fluency and problem-solving, and some college and 
career preparation activities. The program also provided courses to develop skills and build character. Outcomes for participants are 
compared to outcomes for nonparticipants. The study was conducted during 2010-13. The study reports improved outcomes but does 
not adequately control for pre-existing differences between the program and comparison groups. 


Hailey, L. A. (2014). The effects of an after-school program: Changing academic performance and promoting success (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3618240) 


Academic remediation program for students with disabilities 


This quasi-experiment estimates the impact on academic achievement of an afterschool program for students with disabilities in one 
middle school in Georgia. The program provided intense content-area remediation and included differentiated instruction and direct 
instruction. Sessions were held for two hours twice per week, with instruction in mathematics during the first hour and instruction in 
reading during the second hour. Students were divided into small groups with one teacher assigned to every ten students. The study 
was conducted in 2007-08. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Roberts, G. A. (2008). Effect of an after-school remediation program on the academic achievement of students with disabilities 
(Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3324724) 
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Academically-focused program for high school students (South Carolina) 

This quasi-experiment estimates the impact on academic achievement of an academically-focused afterschool program for second year 
high school students from Spartanburg County, South Carolina who scored at or below the 50th percentile on the South Carolina High 
School Assessment Program test. The program offered 12 tutoring sessions in English language arts and 12 in mathematics, each lasting 
30 minutes. Students participated in either the English afterschool program, the mathematics afterschool program, or both based on 
previous test scores. The curriculum was organized into daily lessons, each stressing one skill. The study was conducted in 2011-12. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Fulmer, A. D. (2013). Investigating relationships between academic achievement and after school programs in a high school 
environment (Doctoral dissertation). Retrieved from https://scholarcommons.sc.edu/cgi/viewcontent.cgi?article=3378&context=etd 


Afterschool Literacy Program with the Barton Reading & Spelling System® 

This quasi-experiment estimates the impact on reading comprehension of an afterschool program using the Barton Reading & Spelling 
System®. The program, which used one-to-one tutoring, was offered to students in grades 1-6 who were identified as having reading 
difficulties. The program was held twice a week during the fall and spring semesters for one-hour sessions. Outcomes for students who 
participated in this intervention are compared to outcomes for students who did not participate. The program was offered in one school 
system in Arkansas during 2008-09. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Gearhart, S. (2016). Reading comprehension through incidental learning: Efficacy of an after-school literacy program utilizing the Barton 
Reading and Spelling System® (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 10240947) 


Afterschool math program (Tennessee) 

This quasi-experiment estimates the impact on math achievement of an afterschool mathematics program for students in grades 6-8. 
The program targeted students whose math scores were basic or below basic on the state assessment. The study does not provide 
detail about the content of the program. The study was conducted in two middle schools in Tennessee during 2012-13. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. Eligible outcomes do not meet review requirements, and the measures of effectiveness cannot be attributed to 
the intervention. 


Jarratt, K. (2014). Mathematics achievement outcomes for middle school students attending school-based afterschool mathematics 
programs (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3582791) 


Complementary Assistance Learning Program 

This quasi-experiment estimates the effects of the Complementary Assistance Learning Program (CALP) on reading achievement. CALP 
was an afterschool reading remediation program targeted to eighth graders who had a low score on the state seventh grade reading 
assessment. The study was conducted during 2006-07 at one suburban middle school in southeast Georgia. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Sanford, E. E. (2010). Examining the impact of complementary assistance learning on student CRCT scores (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3397116) 


Gevirtz Homework Project 

This quasi-experiment estimates the impact on effort in reading and math, study skills, and social skills of an afterschool homework help 
program. The program was offered to students in grades 4-6 in three elementary schools in one school district in Southern California. 
The program served students with a range of academic achievement. Sessions were held either three or four days per week for 50 
minutes. A credentialed teacher supervised the sessions, with the help of an assistant; either one or both teachers was bilingual. The 
implementation year is not reported. 
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The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Cosden, M., Morrison, G., Albanese, A. L., & Macias, S. (2001). When homework is not home work: After-school programs for homework 
assistance. Educational Psychologist, 36(3), 211-221. 


Knowledge Points 


This quasi-experiment estimates the impact of afterschool tutoring for students in grades 5-8 offered by Knowledge Points, a private 
tutoring company. Reading, mathematics, and science achievement of students receiving the tutoring are compared to outcomes for 
students from a different school who did not receive tutoring. The study was conducted in Nashville, Tennessee during 2008-09 and 
2009-10. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Sebastian, J. (2013). The impact of an after-school intervention program on academic achievement among middle school students 
(Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3587666) 


Math instruction with Classroom Performance System 


This randomized controlled trial estimates the impact of afterschool mathematics instruction using the Classroom Performance System 
(CPS), in which students use handheld "clickers" to respond to teacher questions. The instruction was offered to eighth grade boys for 
six weeks. Outcomes for program participants are compared to outcomes for a control group, who received direct instruction in math 
and solved problems using pencil and paper. The study was conducted in a school in Georgia. The implementation year is not reported. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups and the 
measures of effectiveness cannot be attributed to the intervention. 


Smart, V. J. (2016). Improving student achievement in after school programs through technology: An action research study (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 10144298) 


Pathways to Success 

This quasi-experiment estimates the impact on mathematics and English language arts achievement of an afterschool tutoring program 
for students in grades 2-6 in one elementary school in New Jersey. Outcomes for program participants are compared to outcomes for 
students who did not participate. The study was conducted in 2003-04. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Smith, T. A. (2005). An examination of the effectiveness of a specific after-school academic intervention program on the success of 
at risk students (Doctoral dissertation). Retrieved from https://scholarship.shu.edu/cgi/viewcontent.cgi?referer=&httpsredir=1&arti- 
cle=1203&context=dissertations 


Project Excel 


This quasi-experiment estimates the impact of Project Excel, an afterschool program that provided math remediation for students in 
grades 2-5 in one school district in New Jersey. The program was fully funded by Title | and employed 15 teachers in two buildings. 


The program was offered once a week for 60 minutes, with 45 minutes of instruction and 15 minutes of performance-based assessment. 


Outcomes for participants (who were eligible due to low-income status and test scores) were compared to outcomes for eligible 
nonparticipants. The study was conducted during 2004-05. 


The study reports negative effects of the program. The study does not meet requirements for a higher evidence tier because it does 
not adequately control for pre-existing differences between the program and comparison groups. 


Lovell, C. (2006). The relationship between participation in extended-day programs and mathematics achievement of Title-I eligible 
participants (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3226843) 
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SmartLinks 


This quasi-experiment estimates the impact on academic achievement and attendance of SmartLinks, an afterschool program for 
eighth graders in Alabama funded by the 21st Century Community Learning Centers program. The program includes remediation, 
academic enrichment activities, and recreation. Outcomes for participants are compared to outcomes for nonparticipants. The study 
was conducted at one school in 2003-04. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Pettway, D. D. (2005). An evaluation of SmartLinks (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database 
(UMI No. 3173506) 


Young Reader's Program 

This quasi-experiment estimates the effects on reading achievement of Young Reader's Program, an afterschool reading intervention 
offered to students reading well below grade level in grades 1-3 in Kansas City, Missouri. This study focused on second grade 
participants in one school in 2001. The program met for three hours once per week for 28 weeks, in groups of between 20 to 25 
students. The program used differentiated instruction to teach reading, writing, vocabulary, spelling, listening, and comprehension 
skills through a combination of skills instruction, high-quality children's literature, storytelling, singing, drawing, movement, and 
manipulative-based instruction. Outcomes for program participants are compared to outcomes for students who did not participate. 


The study reports improved outcomes, but the measures of effectiveness cannot be attributed to the intervention. 


Boulden, W. T. (2006). Evaluation of the Kansas City LULAC National Education Service Center's Young Reader's Program. Children & 
Schools, 28(2), 107-114. 


STUDIES OF ACADEMIC PLUS SOCIAL SUPPORTS PROGRAMS 


Best Foot Forward 


This study uses a randomized design to estimate the effects on attitudes toward school and social and emotional competencies of Best 
Foot Forward, an afterschool recreational program targeted to fifth grade African-American boys identified as aggressive. The program 
consisted of 32 lessons over 12 consecutive weeks. Half of the lessons focused on social skills, and half targeted academic motivation 
skills. The randomized design was compromised because some parents could specify whether their children were in the treatment or 
comparison condition. The implementation year is not reported. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Graham, S., Taylor, A., & Hudley, C. (2015). A motivational intervention for African American boys labeled as aggressive. Urban 
Education, 50(2), 194-224. 


Village Model of Care (Study #1) 


This study uses a randomized design to estimate the effects on academic achievement and school behavior of Village Model of Care, 
a program targeted to sixth grade African American youth. The program included three components: (1) structured group mentoring; (2) 
parental empowerment and support services; and (3) community outreach services. It was implemented with youth and their primary 
caregiver(s) over a school year, with afterschool programming four days per week, scheduled gatherings of students with their families, 
and field trips. The study was conducted with four cohorts of students in two schools. The two schools were randomized either to 
treatment or control. The implementation year is not reported. 


The study reports improved outcomes, but the eligible outcomes do not meet review requirements and the measures of effectiveness 
cannot be attributed to the intervention. 


Hanlon, T. E., Simon, B. D., O’Grady, K. E., Carswell, S. B., & Callaman, J. M. (2009). The effectiveness of an after-school program 
targeting urban African American youth. Education and Urban Society, 42(1), 96-118. 
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Village Model of Care (Study #2) 


This quasi-experiment estimates the impact on school connectedness of Village Model of Care, an afterschool program for students 
aged 11-16 that offered academic assistance, group mentoring, parental support, and community outreach services. The study was 
conducted at two urban middle schools. Outcomes for program participants are compared to outcomes for students who did not 
participate. The implementation year is not reported. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Carswell, S. B., Hanlon, T. E., Watts, A. M. & O'Grady, K. E. (2014). Prevention-related research targeting African American alternative 
education program students. Education and Urban Society, 46(4) 434-449. 


STUDIES OF ARTS PROGRAMS 


Intensive music education program 

This randomized controlled trial estimates the effects on reading and mathematics achievement of an intensive afterschool music 
education program for students in grades 1-8 inspired by El Sistema. The program was offered for two hours after school for 39 weeks. 
Each meeting featured 40 minutes of instruction on an orchestral instrument in a small-group setting and 40 minutes of rehearsal in 

an ensemble comprised of players of similar levels of experience and skill. The study was conducted in one parochial school during 
2010-11, 2011-12, and 2012-13. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Holochwost, S. J., Propper, C. B., Wolf, D. P., Willoughby, M. T., Fisher, K. R., Kolacz, J., ... & Jaffee, S. R. (2017). Music education, academic 
achievement, and executive functions. Psychology of Aesthetics, Creativity, and the Arts, 11(2), 147-166. 


Kuumba Kids 


This quasi-experiment estimates the effects on social and emotional competencies of the Kuumba Kids program. The program, held for 
two hours per week for 16 weeks, offered drama and dance activities and was led by African American artists. The purpose of Kuumba 
Kids was to develop self-esteem, creativity, and problem-solving. The program was offered in a low-income neighborhood in Rochester, 
New York. The mean age of participants was nine years. Outcomes for program participants are compared to outcomes for students 
who did not participate. The implementation year is not reported. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Mason, M. J., & Chuang, S. (2001). Culturally-based after-school arts programming for low-income urban children: Adaptive and 
preventive effects. Journal of Primary Prevention, 22(1), 45-54. 


Music therapy-based intervention program 

This randomized controlled trial estimates the effects on social skills competence of a music therapy-based intervention program. The 
program was offered to children ages 6-11 who attended an urban afterschool program. Participants included a mixture of typically 
developing children and children with generalized social, conduct, and/or behavioral deficits. The five-session program included 
movement to music, drumming, instrument playing, improvisation activities, singing, and music performance combined with poetic 
techniques. Outcomes for program participants are compared to outcomes for students who did not participate. The implementation 
year is not reported. 


The study reports improved outcomes but the measures of effectiveness cannot be attributed to the intervention. 


Gooding, L. F. (2010). The effect of a music therapy-based social skills training program on social competence in children and 
adolescents with social skills deficits (Doctoral dissertation). Retrieved from http://diginole.lib.fsu.edu/islandora/object/fsu%3A254290 
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STUDIES OF CAREER/LEADERSHIP PROGRAMS 


After School Matters 


This quasi-experiment estimates the effects on school attendance, credit accumulation, and high school graduation of Chicago's After 
School Matters (ASM). ASM aimed to increase low-income urban adolescents’ knowledge of workplace culture and improve their soft 
skills through paid apprenticeships. Apprenticeships varied but were guided by instructors who provided information and feedback to 
students regarding workplace standards and culture. Instructors were not certified teachers but had expertise in the field that was a 
focus of the apprenticeship. Outcomes for program participants are compared to outcomes for students who did not participate. The 
study was conducted in 2002-03. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Goerge, R., Cusick, G. R., Wasserman, M., & Gladden, R. M. (2007). After-school programs and academic impact: A study of Chicago's 
After School Matters (Issue Brief #112). Chicago, IL: Chapin Hall Center for Children, University of Chicago. 


STUDIES OF PHYSICAL ACTIVITY/HEALTH PROGRAMS 


Active Winners 


This quasi-experiment estimates the effects of the Active Winners program, which included afterschool and summer fitness activities as 
well as encouragement for families, schools, and communities to engage in fitness activities. The study was implemented with middle 
school students in two rural communities in South Carolina. Outcomes for program participants are compared to outcomes for students 
who did not participate. The implementation year is not reported. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Pate, R. R., Saunders, R. P., Ward, D. S., Felton, G., Trost, S. G., & Dowda, M. (2003). Evaluation of a community-based Intervention to 
promote physical activity in youth: Lessons from Active Winners. American Journal of Health Promotion, 17(3), 171-182. 


Girls on the Run 


This quasi-experiment estimates the effects on physical activity of Girls on the Run. Girls in grades 3-6 participated in a 10-week 
afterschool program that engaged them in running in preparation for a 5K race, team building activities, goal setting, and discussions 
of the media's influence on girls' body image. Outcomes for program participants are compared to outcomes for students who did not 
participate. The study was conducted in 2003-07. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Beller, J. M. (2013). Positive youth development programs and high school girls' psychosocial behaviors (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3601435) 


Kids Living Fit™ 

This quasi-experiment estimates the effects on weight management of Kids Living Fit", a 12-week fitness program held after school 

at four elementary schools. Exercise sessions were generally one hour long, were led by a physical fitness trainer, and included 
activities like aerobic dance, stretching, heart rate monitoring, and yoga. The program also included dietary education presentations by 
registered dietitians. The presentations included using food models to identify appropriate serving sizes, guessing how long it would 
take to burn the extra energy found in larger serving sizes, and working in groups to identify the best choice option from fast food 
menus. Parents were encouraged to attend the four dietary sessions. Outcomes for program participants are compared to outcomes for 
nonparticipants. The study was conducted in 2006. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Speroni, K. G., Earley, C., & Atherton, M. (2007). Evaluating the effectiveness of the Kids Living Fit™ program: A comparative study. 
Journal of School Nursing, 23(6), 329-336. 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 


384 


Martial arts program 

This randomized controlled trial estimates the effects on academic achievement and school attendance of an afterschool martial arts 
program for high school students. The study was conducted in a mid-sized city in northern New York state. Outcomes for program 
participants are compared to outcomes for students who did not participate. The implementation year is not reported. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Blowers, J. G. (2007). Impact of an afterschool martial arts program on at-risk students (Doctoral dissertation). Retrieved from ProQuest 
Dissertations and Theses Database (UMI No. 3257356) 


NutriActive Healthy Experience 

This quasi-experiment estimates the effects of NutriActive Healthy Experience, a four-week afterschool program for students in 
kindergarten and first grade, on students' snack choices. The intervention consisted of a 15-minute walk before school and a 90-minute 
afterschool session with 50-60 minutes of physical activity, a 20-minute classroom lesson on nutrition or body awareness and injury 
prevention, and a nutritious afterschool snack. Parents of participating students received four weekly newsletters outlining the nutrition 
concepts covered each week. The study was conducted at four schools, with two schools receiving the program and two serving as 
comparisons. Outcomes for program participants are compared to outcomes for students who did not participate. The implementation 


year is not reported. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Matvienko, O. (2007). Impact of a nutrition education curriculum on snack choices of children ages six and seven years. Journal of 
Nutrition Education and Behavior, 39(5), 281-285. 


Physical activity program (Georgia) 

This quasi-experiment estimates the effects on cardiovascular fitness of afterschool physical activity programs for fifth graders in five 
Georgia schools. The programs were at least nine weeks long and included at least 13 hours of program time. Outcomes for program 
participants were compared to those for nonparticipants. The implementation year is not reported. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Wilson, R. W. (2016). The effect of school-based physical activity on fifth grade cardiovascular fitness (Doctoral dissertation). Retrieved 
from ProQuest Dissertations and Theses Database (UMI No. 10156429) 


STUDIES OF MULTICOMPONENT PROGRAMS 


21st Century Community Learning Centers program (Texas) 
This quasi-experiment estimates the effects on mathematics and reading achievement of an afterschool program funded by the 21st 
Century Community Learning Centers program. The study was conducted in two middle schools in Texas during 2014-15. Outcomes for 


program participants are compared to outcomes for nonparticipants. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Carr, E. M. (2015). Afterschool program interventions that support the academic achievement, behavior, and engagement of at-risk 
student populations (Doctoral dissertation). University of Houston-Clear Lake, Houston, TX. 


After School Education and Safety 

This quasi-experimental study estimates the effects on mathematics and English/language arts achievement for students in grades 3-5 
of the After School Education and Safety program. The program sought to create a safe afterschool environment for students; provide 
academic assistance with a focus on language arts and math; provide enrichment clubs; increase school attendance; and develop 
positive behavior, social and resiliency skills. The study was conducted in 2005 in one California school district. Outcomes for program 
participants are compared to those for students who had never attended the afterschool programming. 
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The study reports both improved and negative outcomes but does not adequately control for pre-existing differences between the 
program and comparison groups. 


Gorke, G. K. (2006). Academic intervention instructors: Effectiveness of graduate-level college students as intervention tutors in an 
afterschool program (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3227047) 


Afterschool program at the Boys & Girls Club of the Peninsula 


This quasi-experiment estimates the effects on English proficiency for English learners in grades 1-7 attending afterschool programs 
offered by the Boys and Girls Club in San Mateo County, California. The program included an hour of homework help in computer 
classrooms or working with staff and volunteers, followed by enrichment and recreation activities (e.g., open gym time, arts and crafts, 
or enrichment programs). Outcomes for program participants are compared to outcomes for students who did not participate. The study 
was conducted in 2004-08. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. 


London, R., Gurantz, O., & Norman, J. (2011). The effect of afterschool program participation on English language acquisition. Afterschool! 
Matters, Spring, 22-29. 


Afterschool program for Black males in middle school 


This quasi-experiment estimates the effects on academic achievement, social and emotional competencies, and grade promotion of a 
researcher-developed afterschool program aimed at preventing academic failure and juvenile delinquency among middle school at-risk 
Black males. The program had three major components: sports activities, study hall, and social issue discussions. All participants and 
non-participants in the study were recommended by the local school district as being at-risk. Outcomes for program participants are 
compared to outcomes for those who did not participate. The study was conducted in Florida in 1998-2000. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Collum, R. C., Jr. (2004). An investigation of the impact of participation in an afterschool program on the academic achievement and 
social behavior of adolescent Black males defined as at-risk (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses 
Database (UMI No. 3098371) 


Afterschool program operated by EduCare Foundation 


This quasi-experiment estimates the effects on academic achievement, school attendance, and credits earned of an afterschool 
program for high school students. The program offered homework help and enrichment activities. Outcomes for program participants 
are compared to outcomes for students who did not participate. The study was conducted in Southern California in 2014-15. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Educational Research Consultants. (2016). EduCare Foundation: After School Program Report Card for 2014-2015. Retrieved from 
http://www.ercdata.com 


Afterschool programs in South Carolina 


This quasi-experiment estimates the effects on reading and mathematics achievement of afterschool program participation in Greenville 
County, South Carolina. Participants were fourth and fifth graders attending one of 11 Title | schools. The study does not provide detail 
about the content of the program. Outcomes for program participants are compared to outcomes for students who did not participate. 


The study reports both improved and negative outcomes but does not adequately control for pre-existing differences between the 
program and comparison groups. 


Wright, G. B. (2002). The impact of after school programs on students' academic achievement in Title | schools (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3361423) 
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Capital Kids programs 

This quasi-experiment estimates the impact on math and reading achievement of Capital Kids programs, offered in Columbus, Ohio for 
students in elementary and middle grades. The programs provided academic assistance (e.g., homework help, science projects, literacy 
tutoring, and computer math); enrichment activities (e.g., cooking, field trips, drama club, and art projects); skill building units (e.g., anger 
management, alcohol/tobacco/drugs, violence, and peer mediation); safety and wellness (e.g., exercise and recreation); and family 
engagement (e.g., family fun nights, resource referrals, parent-staff conferences, parenting classes, and program newsletters). Out- 
comes for program participants are compared to outcomes for students who did not participate. The study was conducted in 2003-04. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Anderson-Butcher, D., & Cash, S. (n.d.) Capital Kids evaluation: Overall and site specific results for 2003-2004. Columbus, OH: Ohio 
State University. 


Exito 

This quasi-experiment estimates the effects on attendance, promotion, and course grades of Exito, an afterschool program for students 
in grades 9-10 who were identified as at-risk of dropping out of school. The program, offered at one high school in Philadelphia during 
2009-10, took place four days per week. Students received homework help and participated in project-based learning groups. The 


study compares outcomes for program participants to outcomes for a matched group of students from the same school and grade who 
did not attend the program. 


The study reports both improved and negative outcomes but does not adequately control for pre-existing differences between the 
program and comparison groups. 


Hartmann, T., Good, D., & Edmunds, K. (2011). Exito: Keeping high-risk youth on track to graduation through out-of-school time supports. 
Philadelphia: Research for Action. 


Multi-component afterschool program (New Jersey) 


This quasi-experiment estimates the effects of an afterschool program that was implemented in four school-based community learning 
centers in one urban school district in New Jersey. The programs were offered at three elementary schools and one middle school. 
Activities included one-on-one homework help; recreation (gym, arts and crafts, table games, and computer time); and enrichment and 
cultural activities. The study, conducted in 2007-08, compared English language arts outcomes for program participants in grades 4-6 to 
outcomes for students who did not participate. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Delucci, G. R. (2010). An evaluation of an after-school program for low income elementary and middle school students (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3420956) 


Pathways to Progress 

This quasi-experiment estimates the effects on academic achievement and school attendance of Pathways to Progress, an afterschool 
program offered in five elementary schools and three middle/junior high schools in an urban school district in Minnesota from June 
2000 to May 2003. The program offered one hour of academic programming and one hour of recreational or cultural enrichment each 
day. Outcomes for program participants are compared to outcomes for students who did not participate. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Wahlstrom, K., Sheldon, T., & Lewis, A. (2004). Final evaluation report: 21st Century Community Learning Centers Pathways to Progress, 
Saint Paul Public Schools. St. Paul: University of Minnesota Center for Applied Research and Educational Improvement. 


Roosevelt Village Center 


This quasi-experiment estimates the effects on academic achievement, school attendance, arrests, and juvenile delinquency of the 
Roosevelt Village Center, an afterschool program for Oakland, California students in grades 6-12 offered in 2007. The program, held for 
three hours daily after school, offered homework help, cultural enrichment, extracurricular activities, and mentoring by college students. 
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Outcomes for program participants are compared to outcomes for students who did not participate. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Le, T. N., Arifuku, I., Vuong, L., Tran, G., Lustig, D. F., & Zimring, F. (2011). Community mobilization and community-based participatory 
research to prevent youth violence among Asian and immigrant populations. American Journal of Community Psychology, 
48(1-2), 77-88. 


Seventh Grade Center After-School program 

This quasi-experiment estimates the effects on academic achievement of the Seventh Grade Center After-School program, offered 
at one school in Missouri. The program consisted of 45 minutes of homework help, followed by option activities, including open gym, 
swimming, bowling, various clubs, yo-yo club, choir, model-building, student boards, tai-chi, and movie night. Outcomes for program 
participants are compared to outcomes for nonparticipants. The study was conducted during 2000-01. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Hammett, L. A. (2004). A study of the effects of an after-school program on the discipline and the academic achievement of selected 
seventh grade students (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3257307) 


STUDIES OF SPORTS PROGRAMS 


Extracurricular activities and sports (Missouri) 

This quasi-experiment estimates the effects on school engagement of participation in high school sports and extracurricular activities. 
Outcomes for students who participated in sports or activities are compared to outcomes for students who did not participate. The 
study was conducted in one urban school district in Missouri from 2007-13. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. The eligible outcomes do not meet review requirements. 


Martin, C. J. (2015). Student involvement in extracurricular activities and post-secondary education placement (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 10030342) 


School-sponsored sports (Delaware) 

This quasi-experiment estimates the impact of sports participation on GPA and school attendance during the playing season. Outcomes 
for in-season athletes are compared to those for out-of-season athletes. Study participants were students in grades 6-8 in Cape 
Henlopen School District in Delaware during 2012-13. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


O'Grady, S. T. (2013). Analyzing the effects of in-season sports participation on at-risk student-athletes (Doctoral dissertation). Retrieved 
from ProQuest Dissertations and Theses Database (UMI No. 3602814) 


School-sponsored sports (Kansas) 

This quasi-experiment estimates the effects on grade point average of participation in school-sponsored sports, using data for all high 
school students in one suburban Kansas school district. Outcomes for students who participated in at least one sport are compared to 
outcomes for students who did not participate in any sports during 2014-15. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Higgins, J. D. (2016). Get in the game: A study of the relationship between participation in extracurricular activities and academic 
achievement in the Shawnee Mission School District (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses 
Database (UMI No. 10112403) 
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School-sponsored sports (Minnesota) 


This quasi-experiment estimates the effects on school engagement of participation in school sports and/or cheerleading. The 
comparison group did not participate in any sports. The study was conducted in one rural high school in Minnesota in 2001-02. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Dallman, R. (2004). Students’ bond to their school and interscholastic athletics in a consolidated school (Doctoral dissertation). 
Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3163140) 


School-sponsored sports (Nebraska) 


This quasi-experiment estimates the effects of participation in interscholastic sports on GPA, math achievement, and math grades of 
eight graders. The comparison group did not participate in any sports. The study was conducted in one school in Omaha, Nebraska 
in 1998-99. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 


comparison groups. 


Stephens, L. J., & Schaben, L. A. (2002). The effect of interscholastic sports participation on academic achievement of middle level 
school students. NASSP Bulletin, 86(630), 34-41. 


School-sponsored sports (Tennessee) 

This quasi-experiment estimates the effects on academic achievement and school attendance of participation in high school varsity 
sports. Outcomes for students who participated in sports are compared to outcomes for students who did not participate. The study was 
conducted in ten high schools in Tennessee during 2004-05. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Stencel, J. R. (2005). A study of the relationship between interscholastic athletic participation and academic achievement for a 
group of Tennessee high school students (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database 
(UMI No. 3197287) 


School-sponsored sports (Wisconsin) 


This quasi-experiment estimates the effects on academic achievement and school attendance of participation in high school varsity 
sports. Outcomes for students who participated in sports are compared to outcomes for students who did not participate. The study was 
conducted in two suburban high schools in Wisconsin during 1994-98. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Kaufmann, A. M. (2002). Interscholastic sports participation as a predictor of academic success for high school students (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3047511) 


School-sponsored sports (Worcester, Massachusetts) 


This quasi-experiment estimates the impact of sports participation on social and emotional competencies for seventh grade students 
in three middle schools in Worcester, Massachusetts. Outcomes for students who participated in sports are compared to outcomes for 


students who did not participate. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


McHale, J. P., Vinden, P. G., Bush, L., Richer, D., Shaw, D., & Smith, B. (2005). Patterns of personal and social adjustment among 
sport-involved and noninvolved urban middle-school children. Sociology of Sport Journal, 22(2), 119-136. 
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STUDIES OF SCIENCE, TECHNOLOGY, ENGINEERING, MATHEMATICS (STEM PROGRAMS) 


Afterschool science club for middle grades girls 


This quasi-experiment estimates the impact on attitudes toward science of an afterschool science club for seventh and eighth grade 
girls. Seventh grade science club participants worked on hands-on electrical circuit activities, talked with female scientists at a local 
university about their research and how they became involved in science, and participated in monthly field trips to a science center, 
a materials testing company, and scientific laboratories at the university. Eighth grade participants took part in the career/role- 
modeling sessions and the hands-on electrical circuit activities. Outcomes for program participants are compared to outcomes for 
nonparticipants. The study was conducted in 2003-04. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Waldron, A. (2006). A mixed methods evaluation of an afterschool science club's influence on adolescent girls' attitudes toward science 
(Doctoral dissertation). Retrieved from https://www.researchgate.net/publication/34902800_A_mixed_methods_evaluation_of_an_ 
afterschool_science_club's_influence_on_adolescent_girls'_attitudes_toward_science 


Beyond Blackboards 

This quasi-experiment estimates the effects on school engagement of Beyond Blackboards, an afterschool robotics and engineering 
club offered at three middle schools in Cave Creek School District in Texas. The clubs were supported by teachers, who served as 
advisors, as well as engineering faculty, undergraduate students, and engineers from local companies who served as technical mentors 
and role models. Outcomes for program participants are compared to outcomes for nonparticipants. The study was conducted in 
2010-11. 


The study reports improved outcomes but does not adequately control for pre-existing differences between the program and 
comparison groups. The eligible outcomes do not meet review requirements. 


Blanchard, S., Judy, J., Muller, C., Crawford, R. H., Petrosino, A. J., White, C. K., ... & Wood, K. L. (2015). Beyond blackboards: Engaging 
underserved middle school students in engineering. Journal of Pre-college Engineering Education Research, 5(1), 1-14. 


Science Café 


This quasi-experiment estimates the effects on STEM interest and engagement with schooling of Science Café, an afterschool program 
designed to foster high school students' interest in STEM disciplines. Scientists conduct informal early evening presentations in casual 
settings on various topics. High school students run all aspects of the Science Café meetings. The study was conducted in four towns in 
New Mexico in 2010. 


The study reports an improved outcome but does not adequately control for pre-existing differences between the program and 
comparison groups. 


Hall, M. K., Foutz, S., & Mayhew, M. A. (2012). Design and impacts of youth-directed Café Scientifique program. International Journal of 
Science Education, Part B: Communication and Public Engagement, 1-24. 


Science, technology, engineering, and mathematics education programs (New Jersey) 


This quasi-experiment estimates the effects on attitudes toward, interest in, and enjoyment of science of STEM afterschool programs. 
Study participants were in grades 4-6 in three New Jersey school districts. The districts used different approaches to their STEM 
afterschool programming. For example, one district offered a combination of sports science, engineering, and science (Biology, Chemis- 
try, Forensics, Physical Science). Outcomes for programs participants were compared to outcomes for non-participants. Another district 
partnered on its STEM afterschool program with a state-run informal learning institution and included aquatics projects. 


The study reports that there were no statistically significant effects of the program. The study does not meet requirements for a higher 
evidence tier because it does not adequately control for pre-existing differences between the program and comparison groups. 


Smith, C. F. (2015). Transforming attitudes and lives: Liberating African-American elementary and middle school students in out-of- 
school time STEM education (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3684470) 
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APPENDIX EG-3 


STUDIES WITH INSUFFICIENT INFORMATION TO ASSIGN AN EVIDENCE TIER 


About studies with insufficient information to assign an evidence tier 

In some cases, study authors did not report all the information we needed to assign an evidence tier. For example, sometimes they did 
not report information on the statistical significance of a finding or enough information for us to compute the statistical significance 
ourselves. Without information on whether a finding is significant or not, the tier cannot be determined. 


21ST CENTURY COMMUNITY LEARNING CENTERS PROGRAM (SAN FRANCISCO) 


Program type: Multicomponent 

This quasi-experiment examines the effects of afterschool programs operated at four middle schools in San Francisco. The programs, 
funded by the 21st Century Community Learning Centers program, included an academic component (homework time, one-on-one 
tutoring) and enrichment activities such as rock climbing, cooking class, and social activities. Reading and math achievement and 
attendance for participants are compared to outcomes for a matched group of non-participants. The study was conducted during 
1998-99. 


Trousdale, D. (2000). First-year evaluation of an after-school program for middle school youth. ERS Spectrum, 18(3), 3-11. 


GROWING OPTIMALLY, GIRLS IMPROVING HEALTH AND RAISING LEVELS OF SELF-EFFICACY (GO-GIHRLS) 


Program type: Physical activity/health 

The Growing Optimally, Girls Improving Health and Raising Levels of Self-efficacy (GO-GIHRLS) program was an afterschool nutrition 
and exercise program designed to improve nutrition knowledge and increase physical fitness among fourth and fifth grade girls ata 

selective elementary school. Physical activities varied each week but including running, basketball, soccer, dancing, kickball, softball, 


calisthenics, etc. Nutrition classes were taught by an experienced nutrition educator. 


Beckemeyer, P. (2009). An intervention study of pre-teen girls to improve eating behavior and physical activity levels: Growing 
Optimally, Girls Improving Health and Raising Levels of Self-efficacy (GO-GIHRLS) (Doctoral dissertation). DeKalb, IL: Northern 


Illinois University. 


SMART 


Program type: Academic 

The Boys and Girls Clubs SMART program was a multifaceted intervention targeted to second and third grade students with the 

goal reducing their risk of future substance abuse. The SMART Kids prevention curriculum was offered as part of the Boys and Girls 
Club’s afterschool program and was coupled with tutoring and homework help. SMART Teachers linked the Boys and Girls Clubs with 
children’s school and SMART Parents supported parents to help their children succeed in school. 


St. Pierre. T. L., Mark, M. M., Kaltreider, D. L., & Campbell, B. (2001). Boys & Girls Clubs and school collaborations: A longitudinal study 
of a multicomponent substance abuse prevention program for high-risk elementary school children. Journal of Community Psychology, 
29(2), 87-106. 


WAKE COUNTY SUPER OPPORTUNITIES WITH AFTER-SCHOOL RESOURCES 


Program type: Academic 

This quasi-experiment sought to estimate the effects on student academic outcomes of the Wake County Super Opportunities with 
After-School Resources (SOAR) afterschool program. The study examined outcomes for high school students. The program offered 
academic assistance in core subjects, computer skills, Nova NET (an online computer application for skill remediation and recovery of 
high school course credit), SAT prep, and life skills training. Each high school program offered sessions three or four days a week, from 
2 to 3.5 hours per day. 


Nears, K. (2007). The achievement gap: Effects of a resilience-based after school program on indicators of academic achievement. 
(Doctoral dissertation). Retrieved from https://repository.lib.ncsu.edu/handle/1840.16/3402 


AFTERSCHOOL PROGRAMS: A REVIEW OF EVIDENCE UNDER THE EVERY STUDENT SUCCEEDS ACT | PROGRAM SUMMARIES 


392: 


WRITING TO LEARN AFTERSCHOOL PROGRAM 


Program type: Academic 

This quasi-experimental study estimates the effects of "writing to learn strategies" on mathematics, science, and language arts 
outcomes for students in grades 4-7 who scored Partially Proficient on the New Jersey state standardized assessment. Instruction 
in these strategies was offered for 90 minutes after school for 12 weeks during the 2013-14 school year. Comparison students 


received general remediation in an afterschool setting. 


Tarchichi, R. (2017). The impact of mathematics, science and language arts writing to learn strategies on the culture of learning 
in primary and secondary students (Doctoral dissertation). Retrieved from https://rdw.rowan.edu/cgi/viewcontent.cgi?refer- 
er=&httpsredir=1&article=3361&context=etd 
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APPENDIX EG-4 


STUDIES OF PROGRAMS THAT COMBINE AFTERSCHOOL & SUMMER LEARNING ACTIVITIES 


About studies of programs with both afterschool and summer learning components 

This appendix describes studies that 1) examine the effects of year-round programs that include both summer and school year 
activities, where a majority of activities do not occur during the school year; 2) provide required information to assign an ESSA tier; 
and 3) do not separately estimate the effect of the afterschool component. Although these studies of year-round programs are not 
eligible for this review, we present brief summaries here because they may be useful to decision-makers considering year-round 
out-of-school programming. 


STUDIES OF AFTERSCHOOL/SUMMER LEARNING PROGRAMS THAT MEET CAUSE-AND-EFFECT REQUIREMENTS FOR ESSA TIER II 


Extended Day Treatment Program 


This quasi-experiment estimates the effects of the Extended Day Treatment Program (EDTP) on impulse control, prosocial behavior, and 
empathy for students ages 6 to 12 with emotional and behavioral disorders. Children attended the program after school and/or during 
the summer. Outcomes for students who participated in EDTP are compared to outcomes for students who attended the same schools 
as treatment students of similar age, gender, and behavioral issues who did not participate in EDTP. The study found that boys in the 
treatment group showed a greater increase in empathy than boys in the comparison group. No significant group differences were found 
for impulse control or prosocial behavior. 


Wyatt, J. M. (2002). An outcome evaluation of an after-school program for children with emotional and behavioral disorders (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3055298) 


Taking Action Together 

This quasi-experiment examines the effects on BMI, nutrition, and physical activity of Taking Action Together, a community-based 
intervention to reduce the risk of type 2 diabetes mellitus among low-income, high-BMI, 9-10-year-old Black children. Outcomes for 
participating children across two sites were compared to outcomes for comparison group children, who were offered a free week of 
traditional YMCA summer camp. The study found that a larger percentage of children in the treatment decreased BMI z-scores after one 
year, compared to children in the comparison group. The study did not provide results for the nutrition or physical activity outcomes. 


Ritchie, L. D., Sharma, S., Ikeda, J. P., Mitchell, R. A., Raman, A., Green, B. S., ... & Fleming, S. E. (2010). Taking Action Together: A 
YMCA-based protocol to prevent type-2 diabetes in high-BMI inner-city African American children. Trials, 17(1), 60. 


STUDIES OF AFTERSCHOOL/SUMMER LEARNING PROGRAMS THAT MEET CAUSE- AND-EFFECT REQUIREMENTS FOR ESSA TIER IV 


Build IT 


This quasi-experiment examines the effects on attitudes and interests in IT careers of Build IT, an afterschool and summer STEM 
program for middle school girls. The program uses IT professionals to guide girls through developing communications technologies 
using simple programming tools. The study reports that there were no statistically significant effects of the program. 


The study does not meet requirements for a higher evidence tier because it does not adequately control for pre-existing differences 
between the program and comparison groups. 


Koch, M., Georges, A., Gorges, T., & Fujii, R. (2010). Engaging youth with STEM professionals in afterschool programs. Meridian Middle 
School Computer Technologies Journal, 13(1), 1-10. 


Informal Science Education Out-of-School Time Enrichment Programs 

This quasi-experiment examines the effects of participation in science-related out-of-school time enrichment programs on African 
American middle and high school students’ racial identities, attitudes about science, and self-reported science achievement. Students 
participated in year-round science programs with activities both afterschool and over the summer. The study was conducted in an urban 
setting in the Mid-Atlantic region in 2007. The study reports that there were no statistically significant effects of the program. 


The study does not meet requirements for a higher evidence tier because it does not adequately control for pre-existing differences 
between the program and comparison groups and the outcomes do not meet review requirements. 


James, S. M. (2007). Identity and science learning in African American students in informal science education contexts (Doctoral 
dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3300822) 
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Ninth Grade Success Initiative 

This quasi-experiment examines the effects on student credit accumulation and grade promotion of a basic skills program for ninth 
grade students at risk of not earning enough credits for promotion to tenth grade. The self-paced program operated for two hours after 
school and Monday through Thursday over the summer. It used a curriculum focused on the Texas Essential Knowledge Skills that all 
students in the state are expected to learn. The study was conducted in one high school in Texas in 2001-02. The study reports that 
there were no statistically significant effects of the program. The study does not meet requirements for a higher evidence tier because it 


does not adequately control for pre-existing differences between the program and comparison groups. 


Watson, N. (2003). The effects of the ninth grade success initiative at Pearland High School (Doctoral dissertation). Retrieved from 
ProQuest Dissertations and Theses Database (UMI No. 3094018) 


Open Doors Academy 

This quasi-experiment examines the effects of Open Doors Academy on the achievement-related behaviors, developmental assets, 
and academic achievement of middle and high school students in three Ohio school districts. Open Doors Academy is a year-round 
out-of-school enrichment program that includes tutoring/academic support and daily enrichment activities. The study reports improved 
outcomes but does not adequately control for pre-existing differences between the program and comparison groups. 


Grassi, A. M. (2012) Quality after-school programming and its relationship to achievement-related behaviors and academic perfor- 
mance (Doctoral dissertation). Retrieved from https://etd.ohiolink.edu/!etd.send_file?accession=csu1354646131&disposition=inline 


Supplemental Education Services (SES) at one Georgia School 

This quasi-experiment estimated the effects of supplemental education services for at-risk eighth grade students on reading and math 
Georgia Criterion Referenced Competency Test (CRCT) scores, compared to seventh grade non-participants. Tutoring services were 
available afterschool, on Saturdays, and over the summer. The study reports improved outcomes but the outcomes cannot be attributed 
solely to the intervention; grade level is completely confounded with intervention receipt. 


Maxwell, J. B. (2010). The impact of supplemental education services program on middle school at-risk learners’ math and reading 
scores (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses Database (UMI No. 3403821) 
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